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SUMMARY 


The  Advanced  On-the-job  Training  System  (AOTS)  was  an  Air  Staff  directed, 
AFHRL  developed,  prototype  system  which  designed,  developed,  and  tested  a 
proof-of-concept  prototype  AOTS  within  the  operational  environment  of  selected 
work  centers  at  Bergstrom  AFB,  Texas,  and  Ellington  ANGB,  Texas,  from  August 
1985  through  31  July  1989.  The  AOTS  Prime  Item  Specification  for  the 
Evaluation  Subsystem  establishes  the  performance,  design,  development,  and 
test  requirements  for  the  Evaluation  Subsystem.  It  was  prepared  using  MIL- 
STD-490A,  Specification  Practices,  as  a  guide.  The  Evaluation  Specification 
provides  the  support  necessary  to  develop  and  deliver  evaluation  instruments 
as  required  to  achieve  task  proficiency  for  progress  of  individual  airmen 
toward  training  quality  control  and  supports  the  evaluation  of  the  AOTS. 
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PREFACE 


This  paper  was  developed  by  Douglas  Aircraft  Company,  the  AOTS  develop¬ 
ment  contractor,  under  Government  Contract  Number  F33615-C-84-0059.  The 
AFHRL  Work  Unit  Number  for  the  project  is  2557-00-02.  The  primary  office  of 
responsibil ity  for  management  of  the  work  unit  is  the  Air  Force  Human 
Resources  Laboratory,  Training  Systems  Division,  and  the  Air  Force  AOTS 
manager  is  Major  Jack  Blackhurst. 
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1.  SCOPE 

1.1  General .  This  specification  establishes  the  performance, 
design,  development,  and  test  requirements  for  the  Evaluation 
Subsystem  prime  item  of  the  Advanced  On-the-Job  Training  System 
(AOTS) .  It  has  been  prepared  using  MIL-STD-490A,  Specification 
Practices,  as  a  guide.  The  Evaluation  Subsystem  of  the  Advanced 
On-the-Job  Training  System  (AOTS)  provides  the  support  necessary 
to  develop  and  deliver  evaluation  instruments  as  required  to 
achieve  task  proficiency  for  individual  airman  progress  toward 
duty  position  qualification.  It  also  provides  the  application  of 
training  quality  control  and  supports  the  evaluation  of  the  AOTS 
System. 

1.2  Automated  .support  ♦  The  AOTS  Evaluation  Computer  Program 
Configuration  Item  (CPCI)  Development  specification  (70S647413) 
provides  a  description  of  automated  support  for  the  Evaluation 
Subsystem.  Support  will  be  provided  for  identifying  and  main¬ 
taining  evaluation  requirements  for  Air  Force  Specialties  (AFSs) ; 
delivering  evaluation  requirements;  applying  training  quality 
control;  and  evaluating  the  AOTS  System. 
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2 .  APPLICABLE  DOCUMENTS 

2.1  Government  documents.  The  following  government  documents  of 
the  exact  issue  t,nown  form  a  part  of  this  specification  to  the 
extent  specified  herein.  In  the  event  of  conflict  between  the 
documents  referenced  herein  and  the  contents  of  this  specifica¬ 
tion,  the  contents  of  the  specification  shall  be  considered  a  su¬ 
perseding  requirement. 


SPECIFICATIONS: 

70S647100 


70S647200 


70S647400 


70S647500 


STANDARDS : 

MIL-Q— 9858A 
7  Aug  1981 

MIL-S— 52779D 
22  Oct  1983 

MIL-STD— 470A 

3  Jan  1983 

MIL-STD— 48 0A 
12  Apr  1978 

MIL-STD-483A 

4  Jun  1985 


Prime  Item  Development  Specification 
for  Management  Subsystem 

Prime  Item  Development  Specification 
for  Training  Development  and 
Delivery  Subsystem 

Prime  Item  Development  Specification 
for  Computer  Support  Subsystem 

Prime  Item  Development  Specification 
for  Personnel  and  Support  Subsystem 


Quality  Program  Requirements 


Software  Quality  Assurance  Program 
Requirements 

Maintainability  Program  for  Systems 
and  Equipment 

Configuration  Control  -  Engineering 
Changes,  Deviations,  and  Waivers 

Configuration  Management  Practices  for 
Systems,  Equipment,  Munitions  and 
Computer  Programs 


MIL-STD-490A 
4  June  1985 


Specification  Practices 
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MIL-STD-499A 
1  May  1974 

MIL-STD— 78 5B 
15  Sep  1980 

MIL-STD— 8  8  2 A 
1  Apr  1974 

MIL-STD-1472C 
1  Sep  83 


MIL-STD-152 1A 
21  Dec  81 


OTHER  PUBLICATIONS: 

AFR  12-35 
3  Jun  85 

AFR  35-8 

25  Apr  83 

AFR  35-41 

26  Apr  85 

AFR  50-8 
6  Aug  84 

AFR  50-23 
Jun  87 

AFP  50-58 
15  Jul  78 

AFM  50-62 
15  Jan  84 


Engineering  Management 


Reliability  Program  for  Systems  and 
Equipment  Development  and  Production 

System  Safety  Program  for  Systems  and 
Associated  Subsystems  and  Equipment 

Human  Engineering  Design  Criteria  for 
Military  Systems,  Equipment,  and 
Facilities 

Technical  Reviews  and  Audits  for 
Systems,  Equipment  and  Computer 
Programs 


Air  Force  Privacy  Act  Program  (PA) 


Air  Force  Military  Personnel  Testing 
System 

Reserve  Personnel  Policies  and 

Procedures  -  Reserve  Training  (Vol  2) 

Policy  and  Guidance  for  Instructional 
System  Development  (ISD) 

On-The-Job  Training  (PA) 


Handbook  for  Designers  of  Instructional 
Systems  (Vols  1-6) 

Principles  and  Techniques  of  Instruction 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 

2.2  Non-government  documents.  The  following  documents  of  the 
exact  issue  shown  form  a  part  of  this  specification  to  the  extent 
specified  herein.  In  the  event  of  conflict  between  the  documents 
referenced  herein  and  the  contents  of  this  specification,  the 
contents  of  the  specification  shall  be  considered  a  superseding 
requirement . 

OTHER  PUBLICATIONS : 

Structured  Systems  Analysis:  Tools  and  Techniques,  October 
1977;  Gane,  Chris  and  Sarson,  Trish. 
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3 .  REQUIREMENTS 

3.1  Prime  i+am  definition.  This  specification  establishes  the 
requirements  for  the  Evaluation  Subsystem  for  the  Advanced  On- 
the-job  Training  System  (AOTS) .  The  Evaluation  Subsystem  shall 
provide  a  computer-based  development  and  implementation  process, 
within  which  evaluation  materials  and  procedures  can  be 
developed.  The  Evaluation  Subsystem  shall  provide  capabilities 
for  the  development  of  behavioral  objectives  and  the  development 
and  use  of  evaluation  materials  to  assess  task  proficiency  and 
knowledge  of  trainees.  The  Evaluation  Subsystem  shall  manually 
collect  and  analyze  data  that  are  related  to  the  effectiveness 
and  cost  of  job-site  training  and  evaluation  development. 

The  Evaluation  Subsystem  shall  present,  score  and  analyze  tests 
used  for  task  knowledge/performance  evaluations  and  Quality  Con¬ 
trol  evaluations.  It  shall  collect  task  performance  evaluation 
data,  and  generate  all  system  evaluation  reports.  The  subsystem 
shall  consist  of  the  following  four  components: 

a.  Evaluation  Instrument  Management 

b.  Performance  Evaluation 

c.  Training  Quality  Control 

d.  System  Evaluation. 

3.1.1  Prime  it^  fllflgmB.  This  section  incorporates  the 
prime  item  level  functional  schematics.  The  section  shows  top- 
level  functional  flow  diagrams  of  the  configuration  item  and  in¬ 
cludes  functional  flow  diagrams  to  the  level  required  to  identify 
all  essential  functions.  Lower  level  diagrams  giving  further 
definition  and  depth  to  the  operation  of  the  Evaluation  Subsystem 
are  presented  in  Section  10  (Appendix  I) .  Data  dictionaries 
defining  processes,  data  flows,  and  data  stores  are  presented  in 
Section  20  (Appendix  II) .  Figure  1  shows  the  relationship  be¬ 
tween  the  Evaluation  Subsystem  and  the  other  AOTS  subsystems. 
Figure  2  presents  the  four  components  that  make  up  the  AOTS 
Evaluation  Subsystem. 
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Figure  1.  AOTS  Visual  Table  of  Contents 
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Figure  3  shows  the  three  primary  processes  included  in  the 
Evaluation  Instrument  Management  component. 


Figure  3.  Evaluation  Instrument  Management  Component  Overview 
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3.I.X. 2  Performance  Evaluation  Component  overvlgy-  Figure  4 
shows  the  four  primary  processes  included  in  the  Performance 
Evaluation  Component. 


Figure  4.  Performance  Evaluation  Component  Overview 
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3. 1.1. 3  Training  Quality  Control  Component  overview.  Figure  5 
shows  the  two  primary  processes  included  in  the  Training  Quality 
Control  Component. 


Figure  5.  Training  Quality  Control  Component  Overview 
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3. l.i. 4  system  Evaluation  Component  overview.  Figure  6  shows 
the  three  primary  processes  included  in  the  System  Evaluation 
Component . 


Figure  6.  System  Evaluation  Component  Overview 
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3.1.2  Interface  definition.  The  Evaluation  Subsystem  estab¬ 
lishes  functional  interfaces  within  and  outside  of  the  AOTS,  as 
depicted  by  Figure  7.  The  Evaluation  Subsystem  maintains  inter¬ 
faces  with  components  of  other  AOTS  subsystems  and  with  internal 
components  of  the  Evaluation  Subsystem.  Section  10  (Appendix  I) 
depicts  the  functional  interfaces:  (a)  among  the  Evaluation  Sub¬ 
system  and  the  other  subsystems  and  (b)  within  the  Evaluation 
Subsystem  components.  Each  interface  is  discussed  separately. 

3. 1.2.1  interfaces  within  AOTS.  The  Evaluation  Subsystem  in¬ 
terfaces  with  the  following  AOTS  subsystems: 

a .  Management 

b.  Computer  Support 

c.  Personnel  and  Support 

d.  Training  Development  and  Delivery. 

3. 1.2. 1.1  Management  Subsystem  interface .  The  Evaluation  Sub¬ 
system  interfaces  with  the  Management  Subsystem  as  described 
below: 


3. 1.2. 1.1.1  The  Evaluation  Subsystem  shall  provide  the  Manage¬ 
ment  Subsystem  with  the  following: 

a.  Data  which  identify  behavioral  objectives,  tests  and  test 
items  that  apply  to  job-site  tasks 

b.  Data  which  identify  training  and  evaluator  resources 
required  for  job-site  tasks 

c.  Data  which  identify  references  that  apply  to  behavioral  ob¬ 
jectives  and  test  items 

d.  Results  of  knowledge  and  performance  evaluations 

accomplished  for  pre-training,  post-training  and  Quality 
Control  (QC)  purposes 

e.  Data  which  identify  tasks,  evaluators  and  evaluatees  which 
have  been  selected  for  Quality  Control  (QC)  evaluations. 
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Figure  7.  Evaluation  Subsystem  Interface  Overview 


13 


Specification  Number  70S647300B 
Coda  Idant.  Mo.  76301 
Data:  31  October  1989 


3. 1.2. 1.1. 2  The  Management  Subsystem  shall  provide  the  Evalua¬ 
tion  Subsystem  with  the  following: 

a.  Task  analysis  data  for  job-site  tasks 

b.  Training  management  data  for  individuals  operating  under 
the  AOTS 

c.  Evaluation  events  data  for  individuals  and  groups 

d.  Task  and  individual  qualification  factors  for  quality 
control  processing 

e.  Training  management  and  evaluation  events  data  needed  to 
evaluate  system  effectiveness. 

3. 1.2. 1.2  Computer  Support  Subsystem  interface.  The  Computer 
Support  Subsystem  shall  provide  the  hardware,  software  and  com¬ 
munications  necessary  to  support  the  Evaluation  Subsystem. 

3. 1.2. 1.3  Personnel  and  Support  Subsystem  interface.  The  Per¬ 
sonnel  and  Support  Subsystem  shall  identify  personnel  skill  and 
training  requirements  and  provide  for  on-the-job  training 
required  to  accomplish  the  Evaluation  Subsystem  functions.  The 
Personnel  and  Support  Subsystems  shall  also  provide  maintenance 
and  logistics  support,  human  engineering  and  reliability  for  ef¬ 
fective  implementation  and  operation  of  the  Evaluation  Subsystem. 
The  Evaluation  Subsystem  functional  requirements  shall  be  used  to 
help  determine  the  personnel  skill  and  training  requirements  for 
the  Personnel  and  Support  Subsystem. 

3. 1.2. 1.4  Training-Development  and  Delivery  Subsystem  interface. 
The  Evaluation  Subsystem  interfaces  with  the  Training  Development 
and  Delivery  Subsystem  as  described  below: 

3. 1.2. 1.4.1  The  Evaluation  Subsystem  shall  provide  the  Training 
Development  and  Delivery  Subsystem  with  behavioral  objectives 
data  for  the  purpose  of  developing  training  materials. 

3. 1.2. 1.4.2.  The  Training  Development  and  Delivery  Subsystem 
shall  provide  the  Evaluation  Subsystem  with  graphics  for  develop¬ 
ing  test  items  and  administrating  on  line  tests. 

3 . l . 2 . 2  Interfaces  with  sources  external  to  aots.  The  Evalua¬ 
tion  Subsystem  interfaces  with  external  sources  as  described 
below: 


3. 1.2. 2.1  Publications  Management  interface.  An  interface  will 
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be  established  between  the  prototype  AOTS  and  publication  manage¬ 
ment  activities  to  ensure  those  references  containing  pertinent 
task  data,  and  consequently  used  in  the  planning  and  development 
of  behavioral  objectives  and  evaluation  materials,  are  available 
and  maintained  on  a  current  basis.  An  Air  Force  Customer  Account 
Representative  (CAR)  will  interface  with  the  Publicatons  Dis¬ 
tribution  Office  (PDO)  at  Bergstrom  AFB,  Texas  for  ordering  and 
receiving  Air  Force  and  applicable  major  command  publications. 
An  Air  Force  representative  shall  also  interface  with  the  Techni¬ 
cal  Order  Distribution  Office  (T0D0)  at  Bergstrom.  It  is  an¬ 
ticipated  that  access  to  the  Automated  Technical  Order  Main¬ 
tenance  System  (ATOMS)  will  be  required  for  evaluation  instrument 
developers  when  AOTS  becomes  operational  in  the  Air  Force.  Ac¬ 
cordingly,  if  the  ATOMS  is  implemented  at  Bergstrom  AFB  during 
the  development  of  the  prototype  AOTS,  the  establishment  of  an 
interface  will  be  considered  at  that  time. 

3. 1.2. 2. 2.  Operational  Unit  interface.  The  Evaluation  Subsystem 
shall  interface  with  the  operational  units.  The  Evaluation  Sub¬ 
system  shall  provide  the  operational  units  with  evalution  instru¬ 
ments  both  on  line  and  off  line,  test  results,  and  system  effec¬ 
tiveness  reports.  The  operational  units  will  provide  operational 
conditions,  factors  and  feedback  pertaining  to  the  planning, 
development  and  use  of  evaluation  methods  and  materials  which 
derive  from  processes  within  the  Evaluation  Subsystem. 

3. 1.2.3  Evaluation  Subsystem  Component  interface.  The  Evalua¬ 
tion  Subsystem  Components  interface  with  each  other  as  described 
below: 


3. 1.2. 3.1  Evaluation  Instrument  Management  Component  interface. 
The  Evaluation  Instrument  Management  Component  shall  provide  the 
Performance  Evaluation  Component  with  developed  tests  and  test 
items  for  the  purpose  of  administering  and  scoring  evaluations. 

3. 1.2. 3. 2  Performance  Evaluation  Component  interface.  The  Per¬ 
formance  Evaluation  Component  shall  provide  test  scoring  results 
to  the  Evaluation  Instrument  Management  Component  for  the  purpose 
of  analyzing  and  validating  evaluation  materials.  The  Perfor¬ 
mance  Evaluation  Component-  shall  support  the  administration  and 
scoring  of  quality  control  evaluations  for  the  Training  Quality 
Control  Component. 

3. 1.2. 3. 3  Training  Quality  Control  Component  interface.  The 
Training  Quality  Control  Component  shall  identify  qualified 
tasks,  evaluatees  and  evaluators  for  quality  control  evaluation 
events  supported  by  the  Performance  Evaluation  Component. 
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3.1.3  Major  component  list.  \s  mentioned  in  3.1,  the  Evaluation 
Subsystem  shall  consist  of  the  following  components: 

a.  Evaluation  Instrument  Management  Component 

b.  Performance  Evaluation  Component 

c.  Training  Quality  Control  Component 

d.  System  Evaluation  Component. 

3.1.4  Government  furnished  property  list.  The  Instructional 
Support  System  (ISS)  will  be  furnished  by  the  government.  Where 
applicable,  it  will  be  modified  to  meet  requirements  of  the 
Evaluation  Subsystem. 

3.1.5  Government  loaned  property  list.  This  section  is  not  ap¬ 
plicable  to  this  specification. 

3 . 2  Characteristics . 

3.2.1  Performance.  The  Evaluation  Subsystem  shall  provide  the 
evaluation  procedures  and  controls  used  to  effectively  support 
the  implementation  and  operation  of  the  prototype  AOTS.  The 
Evaluation  Subsystem  shall  be  compatible  with  and  supportive  of 
the  other  subsystems  _n  accordance  with  Specification  Documents 
70S647100 ,  70S647200,  70S647400,  and  70S647500.  The  subsystem 

shall  be  user-friendly  and  supportive  of  multiple  ability  levels. 
This  subsystem  shall  be  responsive  to  requirements  and  demands 
imposed  by  the  other  AOTS  subsystems  and  shall  be  incrementally 
developed  to  be  consistent  with  the  phased  build-up  of  the  AOTS. 
MIL-H-46855A,  MIL-Q-9858A,  MIL-S-52779D,  KIL-STD-470A,  MIL-STD- 

480A,  MIL-STD-490A,  MIL-STD-499A,  MIL-STD-785B,  MIL-STD-882A, 
MIL-STD-1472C,  and  MIL-STD-1521A  may  be  used  as  guides  for  the 
development,  testing  and  implementation  of  the  AOTS  Evaluation 
Subsystem  design.  Design  deficiencies  revealed  during  formative 
evaluation  shall  be  corrected  prior  to  installation  of  the  next 
component.  Specific  training  to  be  given  to  system  users  and 
managers  shall  be  determined  by  the  Personnel  and  Support  Subsys¬ 
tem. 


3 -2. 2  Physical  characteristics.  This  section  is  not  applicable 
to  this  specification. 

3.2.3  Reliability.  The  evaluation  instrument  development  tools, 
evaluation  instrument  analyses,  and  system  reports  that  exist  as 
functional  requirements  of  the  Evaluation  Subsystem  shall  be  the 
basis  for  determining  required  reliability  methodology.  Valid 
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and  reliable  methods  and  mechanisms  shall  be  provided  for  produc¬ 
ing,  and  managing  the  delivery  of  all  evaluation  instruments  and 
devices  necessary  to  support  individual  task  certification  for 
all  airmen  assigned  to  designated  AOTS  work  centers  throughout 
the  period  of  the  contract.  Such  methods  will  include  statisti¬ 
cal  tests  of  significance  of  correlations  based  on  test/retest  if 
resources  permit,  inter-item  correlation  (i.e.,  internal 
consistency) ,  and  intra-class  correlations  of  agreement.  Statis¬ 
tical  measures  of  reliability  shall  be  provided  for  all  evalua¬ 
tive  instrumentation.  Report  reliability  shall  test  for  the 
consistency  of  input  and  output  data.  The  reliability  of  instru¬ 
ments  and  reports  will  ensure  quality  assurance  of  system  effec¬ 
tiveness. 

3.2.4  Maintainability.  System  report  generation  that  exists  as 
a  functional  requirement  of  the  Evaluation  Subsystem  will  provide 
the  capability  to  maintain  the  prototype  AOTS  at  an  effective 
level.  Air  Force  personnel  shall  be  trained  in  the  correct  ap¬ 
plication  of  data  analyses,  report  generation,  and  use  of  results 
in  the  maintenance  of  the  prototype  AOTS.  Specific  details  are 
contained  in  the  AOTS  Maintainability  Plan. 

3.2.5  Environmental  conditions.  This  section  is  not  applicable 
to  this  specification. 

3.2.6  Transportability.  This  section  is  not  applicable  to  this 
specification. 

3.3  Design  and  construction. 

3-3*1  Materials,  processes,  and  parts.  This  section  is  not  ap¬ 
plicable  to  this  specification. 

3.3.2  Electromagnetic  radiation.  This  section  is  not  applicable 
to  this  specification. 

3.3.3  Nameplates  and  product  markings.  This  section  is  not  ap¬ 
plicable  to  this  specification. 

3.3.4  workmanship .  This  section  is  not  applicable  to  this 
specification. 

3.3.5  Interchangeability.  This  section  is  not  applicable  to 
this  specification. 

3.3.6  Safety.  This  section  is  not  applicable  to  this  specifica¬ 
tion. 
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3.3.7  Human  performance/human  engineering.  Good  human  factors 
engineering  practices  shall  be  applied  to  the  Evaluation  Subsys¬ 
tem,  to  ensure  optimal  man-machine  interface.  User-friendly 
menus,  prompts,  and  feedbacks  shall  be  employed  to  ensure'  that 
users/operators  will  adequately  understand  the  utility  of 
development  and  analytical  tools,  the  interface  of  evaluation  in¬ 
struments  with  system  components,  and  the  application  of  gen¬ 
erated  reports.  The  Evaluation  Subsystem  shall  be  developed  in 
accordance  with  MIL-H-46855A. 

3.4  Documentation .  This  section  is  not  applicable  to  this 
specification. 

3.5  Logistics . 

3.5.1  Maintenance .  The  reports  that  exist  as  a  functional 
requirement  of  the  Evaluation  Subsystem  shall  form  the  basis  for 
subsystem  maintenance.  The  reports  shall  contain  instrument 
analyses  and  data  concerning  the  use  of  authoring  aids,  and 
development  time.  Air  Force  personnel  shall  be  trained  in  the 
application  of  analyses  and  test  results  to  effect  necessary  sub¬ 
system  and  data  updates.  Application  of  analytical  results  shall 
assure  that  system  effectiveness  is  maintained.  By  correctly  ap¬ 
plying  response  results,  the  system  development  tools,  evaluation 
instruments,  and  the  reports  themselves  can  be  updated  and  kept 
dynamic  in  response  to  trainee/supervisor,  environmental, 
hardware/resources,  software,  and  system  administrative 
policies/personnel  changes. 

3.5.2  Supply.  This  section  is  not  applicable  to  this  specifica¬ 
tion. 

3.5.3  Facilities  and  facility  equipment.  Design  and  construc¬ 
tion  of  real  property  facilities  will  be  accomplished  by  the 
government . 

3.6  Personnel  and  training. 

3.6.1  Personnel .  This  section  is  not  applicable  to  this 

specification. 

3.6.2  Training.  This  section  is  not  applicable  to  this 

specification . 

3.7  Malor  component  characteristics.  The  Evaluation  Subsystem 
shall  be  capable  of  providing  assessment  of  trainee  task 
proficiency  and  task  requisite  knowledge.  The  system  shall  col¬ 
lect  and  analyze  data  related  to  the  training  efficiency  and  cost 
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effectiveness  of  AOTS  job  site  training.  The  component  charac¬ 
teristics  are  specified  below  and  elaborated  in  Appendix  I  and 
Appendix  II.  Processes  performed,  procedures  followed  and 
products  produced  shall  be  in  accordance  with  AFR  12-35,  AFR  35- 
8,  AFR  35-41,  AFR  50-8,  AFR  50-23,  AFP  50-58,  and  AFM  50-62. 

.7.1  Evaluation  Instrument  Management  Component  requirements ♦ 
The  Evaluation  Instrument  Management  Component  shall  provide 
valid  and  reliable  methodologies  and  mechanisms  for  producing, 
maintaining,  and  managing  the  evaluation  instruments  and  devices 
that  support  individual  job-site  task  performance  evaluation  and 
task  certification.  The  primary  processess  required  to  ac¬ 
complish  the  functions  of  the  Evaluation  Instrument  Management 
Component  include: 

a.  Maintain  Behavioral  Objectives  Data 

b.  Plan  Evaluation  Materials 

c.  Maintain  Evaluation  Materials. 

3. 7. 1.1  Maintain  bghavjprfrl  Object iveg  data-  The  system  shall 
provide  the  capability  to  select  any  task  for  behavioral  objec¬ 
tive  development  or  revision.  Up  to  six  behavioral  objectives 
may  be  developed  for  each  task  and  subtask.  Each  behavioral  ob¬ 
jective  statement  shall  consist  of: 

a.  An  explicitly  stated  behavior 

b.  The  operational  conditions  and  parameters 

c.  The  standards  or  criteria  used  to  demonstrate  trainee 
mastery. 

3. 7. 1.2  Plan  evaluation  materials.  Manual  or  automated  tools 
and  methodologies  shall  be  available  to  analyze  task  performance 
and  proficiency  data,  to  specify  evaluation  strategies,  and  to 
determine  task  performance  evaluation  resource  requirements. 
Procedures  and  mechanisms  provided  by  the  contractor  shall  be 
capable  of  producing  operational  products  that  reflect  Air  Force 
Instructional  System  Development  (ISD)  policy. 

3. 7. 1.2.1  Review/analvze  task  information  .  The  system  shall, 
provide  the  capability  to  examine  available  performance  and 
proficiency  data  and  consider  operational  information  to  develop 
initial  estimates  of  evaluation  parameters.  Estimates  shall  be 
derived  from  primarily  manual  means  which  allow  evaluation 
materials  planner  to: 
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a.  Identify  task  skill  and  knowledge  requirements 

b.  Identify  task  performance  literacy  requirements 

c.  Identify  influencing  environmental  conditions 

d.  Estimate  task  performance  variance  across  work  centers 

e.  Establish  probability  of  change  within  work  centers. 

3. 7. 1.2. 2  Determine  evaluation  strategies  &  resource  require¬ 
ments  .  Evaluation  materials  developers  shall  manually  select 
and  rank  evaluation  strategies.  Selections  and  ranking  decisions 
shall  be  based  on  estimates  of  evaluation  parameters,  resource 
requirements  versus  resource  availability,  relative  cost  of 
strategies,  and  task  priority.  Developers  will  manually: 

a.  Select  candidate  strategies 

b.  Establish  candidate's  resource  requirements 

c.  Compare  resource  requirements  to  availability 

d.  Determine  resource  deficiencies 

e.  Rank  strategies 

f.  Accept  or  override  ranking  of  strategies 

g.  Establish  resource  requirements. 

The  strategies  shall  support  the  direct  observation  of  airmen 
task  performance  for  certification.  In  instances  where  direct 
observation  of  a  task  is  difficult  because  it  is  rarely  performed 
in  the  operational  environment,  but  the  task  is  critical  in  na¬ 
ture,  alternates  to  direct  observation  will  be  specified.  Deter¬ 
mining  resource  and  logistics  requirements  is  part  of  determining 
evaluation  strategies.  Adequate  strategies  with  low  resource 
requirements  shall  be  given  priority.  The  system  shall  have  the 
capability  to  incorporate  new  evaluation  strategies  that  may  be¬ 
come  available  during  the  contract  period.  New  evaluation 
strategies  shall  be  studied,  and  recommendations  for  incorpora¬ 
tion  into  the  Evaluation  Subsystem  shall  be  made,  based  on  pre¬ 
planned  product  improvements. 

3 *7* 1,3  Maintain  Evaluation  Materials.  Processes  used  to  main¬ 
tain  the  evaluation  material  shall  enable  the  development  of  in- 
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dividual  test  items  and  tests  for  terminal  and  supporting  objec¬ 
tives.  The  system  shall  allow  the  capability  to  develop  oral 
test  guides,  performance  evaluation  checklists,  and  knowledge 
items  (i.e.,  multiple  choice,  list  multiple  choice,  true/false, 
matching,  touch,  sequential  touch,  and  constructed  response  test 
items)  . 

Processes  used  to  maintain  the  evaluation  materials  shall  allow 
developers  to: 

a.  Reference  test  items  to  one  or  more  behavioral  objectives 

b.  Reference  test  items  to  one  or  more  publications 

c.  Develop/revise  oral  test  guides  (OTGs) 

d.  Develop/ revise  performance  evaluation  checklists  (PECs) 

e.  Develop/ revise  knowledge  test  items  (e.g.  true/ false 

multiple  choice,  matching,  constructed  response,  touch) 

f.  Incorporate  graphics  in  knowledge  test  item  stems 

g.  Select  test  items  for  tests 

h.  Cross-reference  (code)  items  to  tests 

i.  Specify  test  formats 

j .  Specify  test  parameters 

k.  Specify  test  instructions 

l.  Develop  a  primary  and  at  least  two  alternate  knowledge 
tests  for  each  objective 

m.  Select  alternate  tests  at  random  for  presentation  after  a 
trainee  has  failed  the  primary  test 

n.  Analyze  evaluation  materials 

o.  Validate  evaluation  materials. 

3. 7. 1.3.1  Maintain  Test  Item  Bank.  The  test  items  shall  be 
stored  in  a  Test  Item  Bank  data  store.  The  criterion  for  initial 
acceptability  of  a  test  item  shall  be  that  there  is  agreement 
among  all  assigned  functional -area  subject  matter  experts  (SMEs) 
that  the  item  addresses  what  it  purports  to  address.  There  shall 
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also  be  a  statistically  significant  agreement  .that  the  item  is 
formulated  to  measure  either  performance  in  an  operational  set¬ 
ting  (i.e.,  performance  measures)  or  performance-oriented 
knowledge  (i.e.,  knowledge  items). 

The  contents  of  the  test  item  bank  shall  be  cross-referenced  to 
products  such  as  task  or  subtask  identification  numbers ,  accom¬ 
panying  graphics  and  tests.  All  test  items  shall  be  keyed  to  ap¬ 
proved  terminal  or  supporting  objectives.  Maintaining  the  test 
item  bank  shall  be  supported  by  additional  processes  located  in 
the  process,  Maintain  Tests.  These  processes  shall  allow  test 
developers  to  construct  performance  tests,  consisting  of  oral 
test  guides  (OTG)  and  performance  evaluation  checklists  (PECs) 
and  knowledge  tests  consisting  of  true/false,  multiple  choice, 
matching,  and/or  other  test  items  selected  from  the  data  store. 

3. '7. 1.3. 2  Maintain  Tests.  The  system  shall  provide  test 
developers  with  the  capability  to  develop,  store,  analyze  and 
validate  knowledge  and  performance  tests. 

3. 7. 1.3. 2.1  Develop/Revise  Tests.  The  system  shall  provide 
capabilities  to  format,  edit  and  expand  component  parts  of 
evaluation  modules.  Evaluation  modules  will  consist  of  the  com¬ 
plete  set  of  evaluation  materials  developed  to  evaluate  task  per¬ 
formance  and  task  requisite  knowledge  for  a  particular  AFS  task. 
Modular  evaluation  materials  will  be  developed  that  validly 
assess  airman  task  competencies  for  certification,  recertifica¬ 
tion,  and  training  quality  control. 

3. 7. 1.3. 2. 2  Analyze  evaluation  materials.  The  system  shall 
analyze  evaluation  materials  which  have  been  developed  and  imple¬ 
mented.  Data  to  be  analyzed  shall  be  consolidated  and  sorted  to 
support  a  pre-determined  number  of  samples  for  each  test.  The 
Evaluation  Subsystem  shall  support  automatic  and  requested 
analyses.  Analysis  results  shall  be  printed  in  a  form  suitable 
for  use  in  validating  the  evaluation  materials. 

Wie  system  shall  support  the  analysis  of  a  given  test  and  its 
items  using  one  or  more  of  the  following  criteria: 

a.  samples  administered  between  specified  dates 

b.  samples  passed 

c.  " samples  failed 

d.  all  samples. 
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3. 7. 1.3. 2. 3  Validate  evaluation  materials.  The  system  shall 
provide  hard  copy  printouts  of  knowledge  test  items  and  perfor¬ 
mance  measure  analyses  to  support  the  validation  of  evaluation 
modules  against  a  criterion  of  job  performance  in  an  operational 
setting,  in  accordance  with  the  Air  Force  ISD  policy.  The  system 
shall  provide  the  capabilities  to  identify  materials  requiring 
changes  and  to  revise  those  materials  as  necessary. 

3.7.2  Performance  evaluation  component  requirements  .  The  Per¬ 
formance  Evaluation  Component  shall  provide  capabilities  to  ad¬ 
minister  and  score  requisite  knowledge  tests  and  performance 
tests.  The  Performance  Evaluation  Component  shall  collect  task 
evaluation  results,  for  use  both  in  assessing  individual  task 
performance  and  in  determining  the  adequacy  of  training.  This 
component  shall  support  pre-training,  post-training  and  quality 
control  evaluations.  The  collection  and  recording  of  all  evalua¬ 
tion  results  shall  be  possible  by  online  procedures  with  interac¬ 
tive  computer  terminals,  and  off-line  procedures,  utilizing 
keyboards  or  machine  readable  forms  and  optical  mark  readers. 
The  system  shall  enable  supervisors  and  evaluators  to: 

a.  Administer  requisite  knowledge  tests  on  line  and  off  line 

b.  Test  task  mastery  by  observation  of  task  performance  or 
product  evaluation  off  line 

c.  Score  tests  and  collect/maintain  evaluation  results 

d.  Maintain  accountability  of  materials  printed  for  offline 
use. 

3 •?. 2.1  Administer  knowledge  tests  on  line.  On-line  knowledge 
testing  shall  present  test  items,  accept  trainee's  selected 
answers,  and  allow  the  trainee  to  review  and  change  the  selected 
answers  before  the  test  is  scored.  The  system  shall  have  the 
capability  to  provide  immediate  feedback  to  a  trainee  taking  an 
on  line  knowledge  test  (i.e.,  score  the  knowledge  test,  display 
any  missed  items,  and  indicate  the  correct  answer  for  missed 
items  immediately  after  a  trainee  completes  a  test) . 

3-7*2-2  Administer  evaluations  off  line.  The  system  shall  be 
able  to  print  up  to  75  copies  of  an  off-line  test  in  response  to 
a  single  authorized  request.  The  system  shall  print  the  identi¬ 
cal  test  control  number  on  each  copy  printed  for  a  respective 
print  request  (i.e.,  if  15  copies  are  requested,  15  identical 
copies  will  be  printed  and  all  15  copies  will  have  the  same  test 
control  number) .  The  system  shall  be  able  to  print  a  knowledge 
test  answer  key  if  the  key  is  requested  at  the  time  that  the 
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knowledge  test  is  printed.  No  answer  key  shall  be  available  or 
printed  for  a  performance  test. 

3. 7. 2. 3  Score  tests  and  provide  feedback.  The  system  shall  be 
able  to  score  tests  presented  on  line  or  off  line  and  report  the 
test  results  to  designated  subcomponents  or  processes.  Off-line 
test  results  shall  be  entered  with  a  key  board  or  an  optical  mark 
reader  scan  sheet.  The  system  shall  check  off-line  knowledge 
test  results  against  the  knowledge  test  answer  key  files.  The 
system  shall  accept  pass/ fail  results  for  each  performance  step 
and  the  performance  test  as  a  whole. 

3. 7. 2. 4  Account. for  off-line  evaluation  materials.  The  system 
shall  enable  accountability  of  evaluation  materials  which  are 
printed  for  off  line  administration. 

3.7.3  Training,  quality  control  component  requirements.  The 
Training  Quality  Control  Component  shall  provide  task  performance 
evaluations  to  assess  effectiveness  of  AOTS  training.  This  third- 
party  evaluation  shall  allow  for  evaluation  by  individuals  inde¬ 
pendent  of  the  trainer/ trainee  process. 

3. 7. 3.1  Select._QC  Evaluation  candidates.  The  periodic  selec¬ 
tions  for  a  quality  control  evaluation  event  shall  include  the 
task  to  be  evaluated,  the  airman  to  be  evaluated,  and  two  exter¬ 
nal  evaluators.  An  automatic  process  shall  identify  the 
evaluatee  and  external  evaluator  candidates  for  each  task  iden¬ 
tified  for  quality  control  task  evaluation.  The  process  by  which 
the  system  identifies  tasks,  evaluatees,  and  external  evaluators 
for  quality  control  evaluation  events  shall  recycle  until  a 
predetermined  number  of  QC  evaluation  matches  have  been  iden¬ 
tified.  The  subsystem  shall  notify  the  QC  Administrators, 
through  the  Training  Management  Component,  of  tasks,  evaluatees, 
and  evaluators  identified  as  candidates  for  training  quality  con¬ 
trol.  The  subsystem  shall  also  support  manual  selections  of 
tasks,  evalauatees  and  evaluators  for  quality  control  evalua¬ 
tions  . 


3. 7. 3. 2  Initiate  quality  control  events.  The  system  shall  al¬ 
low  QC  Aministrators  to  initiate  quality  control  evaluation 
events.  The  actual  scheduling  of  QC  events  shall  take  place  in 
the  Management  Subsystem;  data  to  initiate  each  QC  evaluation 
shall  result  from  the  Training  Quality  Control  Component  of  the 
Evaluation  Subsystem,  or  shall  be  manually  determined  by  the  QC 
Administrator. 

3 *7. 3. 3  Follow  up  quality  control  evaluation  event  .  QC  evalua¬ 
tions  shall  be  administered  and  scored  using  the  same  processes 
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as  for  task  performance  evaluations,  accomplished  within  the  Per¬ 
formance  Evaluation  Component.  The  Evaluation  Subsystem  shall 
provide  a  capability  to  enter  quality  control  evaluation  results 
into  the  computer  support  system  by  means  of  appropriate  off-line 
data  entry  devices,  such  as  optical  mark  readers  or  keyboards. 
When  a  QC  evaluation  is  failed,  a  notice  to  the  commander  recom¬ 
mending  decertification  on  the  task  will  be  generated.  Records 
of  quality  control  evaluation  events,  results  and  decertification 
actions  shall  be  maintained  for  system  report  generation,  as 
required.  These  summary  reports  of  training  quality  control  ac¬ 
tivities  will  be  an  additional  source  of  data  beyond  the 
notification  of  decertification  recommendations  provided  to  the 
appropriate  commander,  workcenter  supervisor,  and  unit  training 
manager. 

3.7.4  System  evaluation  component  requirements.  The  System 
Evaluation  Component  of  the  Evaluation  Subsystem  shall  collect, 
maintain,  and  report  data  regarding  the  performance  of  the 
prototype  AOTS  in  meeting  system  goals  for  training  quality  and 
task  performance.  The  reports  shall  include  standard  reports 
generated  at  periodic  intervals  for  prespecified  report 
recipients,  and  ad  hoc  reports  generated  on  demand  by  authorized 
report  recipients  to  address  specific  AOTS  performance  aspects. 

3. 7. 4.1  Consolidate/ sort  reports  data.  The  procedures  used  in 
the  generation  of  both  the  standard  and  the  ad  hoc  reports  shall 
include  sorting  capabilities,  and  a  means  of  specifying  unique 
report  formats,  as  required.  System  historical  data  shall  be 
maintained,  so  that  aggregate  data  can  be  reported  by  work  cen¬ 
ter,  unit,  base,  or  trainer/ evaluator.  Both  standard  reports  and 
ad  hffiS  reports  shall  include  data  summaries  and  statistical 
analyses,  as  required.  The  reports  provided  shall  be  suitable 
for  utilization  at  work  center,  unit,  base,  MAJCOM,  and  Air  staff 
levels. 

3. 7. 4. 2  Calculate  data  and  generate  reports.  The  system  shall 
provide  statistical  summarization  for  both  standard  and  ad  hoc 
reports. 

3. 7. 4. 2.1  Standard  reports.  The  standard  reports  shall  be  gen¬ 
erated  by  the  system,  based  on  menu-driven  options  keyed  to  user 
data  needs  requirements.  Summaries  which  identify  existing  or 
potential  training  problems  affecting  mission  readiness  shall  be 
provided.  The  primary  focus  of  these  summaries  and  reports  shall 
be  the  training  management  requirements  of  the  operational  work 
centers  and  units.  These  management  information  reports  shall 
furnish  up-to-date  information  on  training  effectiveness.  Such 
reports  shall  include  the  following  type  of  information: 
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a.  Position  qualification  status 

b.  Impact  of  Personnel  Loss 

c .  Workcenter  Coverage 

d.  Upgrade  Training  Summaries 

e.  Evaluator  Performance  Data/Summaries 

f.  Trainer  Performance 

g.  Training  Event  Summaries 

h.  CDC  Status 

i.  Recurring  Training  Requirements. 

3. 7. 4. 2. 2  Ad  Hoc  reports  .  The  system  shall  allow  the  user  to 
request  a  user-defined  data  analysis  and  formatted  report  when 
standard  reports  do  not  meet  the  need  and  data  are  available  in 
the  AOTS  data  stores  to  support  the  request.  The  system  shall  be 
flexible  and,  therefore,  capable  of  generating  a  variety  of  ad 
hoc  reports  through  an  automated  process  available  to  system  ad¬ 
ministrators. 

3.8  Precedence.  When  the  application  of  the  requirements  of  the 
AOTS  contract,  this  specification,  or  other  applicable  documents, 
are  in  conflict,  the  following  order  of  precedence  shall  apply: 

a.  AOTS  Contract.  The  terms  of  the  contract  shall  have  prece¬ 
dence  over  all  other  documents. 

b.  This  specification.  For  the  performance  and  design 
requirements  of  the  Evaluation  Subsystem,  this  specifica¬ 
tion  shall,  with  the  exception  of  the  AOTS  contract,  take 
precedence  over  all  other  program  documents  and  applicable 
documents . 

c.  Other  applicable  documents.  Other  documents  referenced 
within  this  specification  shall  have  precedence,  unless 
they  conflict  with  the  AOTS  contract  or  this  specification. 
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4.  QUALITY  ASSURANCE  PROVISIONS 

4.1  General .  Sections  4.1.1  and  4.2  identify  and  describe  the 
Quality  Assurance  provisions  applicable  to  the  Evaluation  Subsys¬ 
tem  of  the  prototype  AOTS.  These  sections  document  the  practices 
which  shall  be  employed  by  the  contractor  to  ensure  adequate  con¬ 
trol  and  to’  establish  conformance  with  the  SOW  and  specifica¬ 
tions.  Quality  Assurance  testing  shall  take  place  at  Bergstrom 
AFB,  TX.  The  contractor  shall  be  responsible  for  conducting  the 
Evaluation  Subsystem  Tests. 

4.2  Quality  conformance  inspections.  Verification  of  perfor¬ 
mance  with  each  requirement  in  Section  3  shall  be  the  respon¬ 
sibility  of  the  contractor.  The  methods  used  to  verify  these 
requirements  shall  be  as  specified  in  Table  I. 
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TABLE  I  QUALITY  CONFORMANCE  INSPECTION 


SECTION 

N/A 

INSPECT 

ANALYZE 

TEST  | 

DEMO 

3.1 


3.1.1 


3.1. 1.1 


3.1. 1.2 


3.1. 1.3 


3.1. 1.4 


3.1.2 


3.1.2.1 


3. 1.2.1. 1 


3.1.2.1.1.1 


3. 1 .2. 1.1.2 


3. 1.2. 1.2 


3:1.2. 1.3 


3.1.2.1.4 


3. 1.2. 1.4.1 


3. 1.2. 1.4.2 


3. 1.2.2 


3.1.2.2.1 


3.1. 2.2.2 


3. 1.2.3 


3.1. 2.3.1 


3.1. 2.3.2 


3.1. 2.3.3 


3.1.3 


3.1.4 


3.1.5 


3.2 


3.2.1 


3.2.2 
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TABLE  I  QUALITY  CONFORMANCE  INSPECTION  Cont. 


30 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


5.  PREPARATION  FOR  DELIVERY 

5.1  General .  This  section  is  not  applicable  to  this  specifica¬ 
tion. 

5.2  Preservation  and  packaging.  This  section  is  not  applicable 
to  this  specification. 

5.3  Packing.  This  section  is  not  applicable  to  this  specifica¬ 
tion. 

5.4  Marking  for  shipping.  This  section  is  not  applicable  to 
this  specification. 
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6 .  NOTES 

6.1  Intended  use.  This  specification  is  to  be  used  in  the  es¬ 
tablishment  of  uniform  practices  to  ensure  the  inclusion  of  es¬ 
sential  requirements  for  the  development  and  implementation  of 
the  Evaluation  Subsystem  for  the  prototype  AOTS. 

6.2  Changes  from  previous  issue.  Asterisks  or  vertical  lines 
are  not  used  in  this  revision  to  identify  changes  with  respect  to 
the  previous  issue  due  to  the  extensiveness  of  the  changes. 
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SECTION  10,  APPENDIX  I 


10.  DATA  FLOW  DIAGRAMS 

10.1  Scope.  This  appendix  establishes  the  design  and  the  inter¬ 
face  requirements  for  the  Evaluation  Subsystem.  A  structured 
system  analysis  approach  has  been  employed,  using  data  flow 
diagrams  (DFDs)  to  illustrate  the  design. 

10.2  Data  flow  diagram  symbols.  The  following  describes  and  il¬ 
lustrates  the  DFD  sysbols  used: 

a.  Square:  an  external  entity  symbol.  A  source  and/or 
destination  of  data  outside  the  subsystem,  component,  or 
process  being  described.  Numbered  references  of  the  exter¬ 
nal  entity  are  contained  in  the  top  left  corner  of  the 
square.  Each  defined  external  entity  is  shown  in  10.3. 
Entities  external  to  AOTS  are  numbered  according  to  the 
convention  established  in  the  Management  Subsystem 
Specification,  70S5647100. 

b.  Arrow:  data  flow  symbol.  A  pathway  along  which  data  moves 
into,  around,  and  out  of  the  subsystem  or  component.  Each 
data  flow  is  labeled.  Definitions  for  data  flows  are  or¬ 
ganized  alphabetically  by  component  in  Appendix  II. 

c.  Rounded  rectangle:  a  process  symbol.  A  function  of  the 
subsystem  or  component  which  logically  "trans forms"  data. 
The  processes  are  labeled  numerically,  beginning  with  "1M , 
on  each  data  flow  diagram.  The  numbering  system  follows  a 
logical  decimal  scheme  to  trace  from  one  process  to 
another.  For  example.  Process  1,  Develop  Behavioral  Objec¬ 
tive  Data  of  DFD  2.1.1  (Maintain  Behavioral  Objectives 
Data)  can  also  be  referred  to  as  process  2. 1.1.1.  Defini¬ 
tions  and  descriptions  for  processes  are  organized  numeri¬ 
cally  by  component  in  Appendix  II.  An  asterisk  located  to 
the  right  of .  the  number  (e.g.,  1*)  indicates  that  at  least 
part  of  the  process  is  manual.  The  data  dictionary  entries 
for  the  processes  marked  with  an  asterisk  identify  the 
manual  part(s)  of  the  processes. 

d.  Open-ended  rectangle:  a  data  store  symbol.  A  place  in  the 
subsystem  or  component  where  data  are  stored  in  some  way. 
The  numbering  scheme  for  identifying  the  data  stores  is 
shown  in  10.3.  Definitions  and  contents  of  data  stores  are 
given  in  numerical  order  by  component  in  Appendix  II. 
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10.3  APTS  Evaluation  subsystem  pfd  numbering. 

a.  Data  Stores 

El  Consolidated  Behavioral  Objectives  Data 

E2  Bad  Verb  List 

E3  Test  Item  Bank 

E4  Consolidated  Test  Data 

E5  Consolidated  Log  Records 

E6  Consolidated  Test  Score  Data 

E7  Test  Key  File 

E8  Quality  Control  (QC)  Log 

E9  Consolidated  Training  Reports  Data 

E10  Training  Reports 

b.  External  Entities  Within  AOTS 

1.1  Training  Requirements  Management  Component 

1.2  Airman  Training  Management  Component 

1.3  Resources  Inventories  and  Event  Schedules  Management 
Component 

3.0  Computer  Support  Subsystem 

4.0  Personnel  and  Support  Subsystem 
5 . 1  Training  Development  Component 

c.  External  to  AOTS 

2  Publications  Management 

5  Operational  Units  ' 
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DFD  2.1.3,  Maintain  Evaluation  Materials 
( Spec  Para  3.7.1.  ) 
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DFD  2. 1.3.1,  Maintain  Test  Item  Bank 
( Spec  Para  3.7. 1  ) 
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DFD  2.2,  Evaluate  Performance 
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SECTION  20,  APPENDIX  II 
20.  DATA  DICTIONARIES 

20.1  Scope.  This  appendix  contains  data  dictionaries  providing 
more  detailed  explanations  of  the  processes,  data  flows,  and  data 
stores.  The  material  is  as  follows: 

a.  Data  processes  -  The  highest  level  processes  were  described 
in  the  text  of  Section  3.7.  The  lower  level  processes, 
which  were  broken  down  in  Appendix  I,  are  described  here. 
They  are  numerically  organized  by  component  and  include  the 
process  name,  the  level  of  automation  involved,  description 
of  the  process,  and  its  structure.  The  levels  of  automa¬ 
tion  are  defined  as  follows: 

(1)  Manual  process  -  A  process  performed  either  partially 
or  totally  off  line.  For  example,  the  process 
"validate  evaluation  modules"  is  performed  off  line  by 
reviewing  a  hardcopy  print  of  test  item  analysis. 

(2)  Automated  process  -  A  process  performed  by  user  inter¬ 
action  with  the  system  using  a  keyboard,  touch  pad, 
and/or  running  a  form  through  an  optical  mark  reader. 
For  example,  entering  the  text,  alternatives,  and 
designating  the  correct  response  for  a  question. 

(3)  Automatic  process  -  A  process  performed  entirely  by 
the  computer  once  the  proper  commands  have  been 
entered.  For  example,  "access  control"  when  a  user 
logs  on  to  the  system. 

b.  Data  flows  -  Contains  a  description  giving  depth  to  the 
data  flow  lines  on  each  DFD.  They  are  alphabetically  or¬ 
ganized  by  component.  E  :h  data  flow  explanation  contains 
the  data  source,  data  destination,  and  structure  when 
known. 

c.  Data  stores  -  Contains  a  description  of  the  data  contained 
within  the  data  store.  They  are  numerically  organized  by 
component.  Each  data  store  explanation  also  contains  the 
data  sources,  data  destinations,  and  the  structure. 
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The  Data 

20.2 

20.2.1 

20.2.2 


20.3 

20.3.1 

20.3.2 

20.4 

20.4.1 

20.4.2 

20.5 

20.5.1 

20.5.2 


Dictionary  is  organized  as  follows: 

Evaluation  Instrument  Management  Component 
Processes . 

Evaluation  Instrument  Management  Component  Data 
Flows . 

Evaluation  Instrument  Management  Component  Data 
Stores . 

Performance  Evaluation  Component  Processes. 
Performance  Evaluation  Component  Data  Flows. 
Performance  Evaluation  Component  Data  Stores. 

Training  Quality  Control  Component  Processes. 
Training  Quality  Control  Component  Data  Flows. 
Training  Quality  Control  Component  Data  Stores. 

System  Evaluation  Component  Processes. 

System  Evaluation  Component  Data  Flows. 

System  Evaluation  Component  Data  Stores. 
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Specification  Number  70S647300B 
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PROCESS:  2.1/  Manage  Evaluation  Instruments 

LEVEL  OF  AUTOMATION:  Manual,  automated,  automatic 


DESCRIPTION:  This  process  shall  enable  authorized  personnel  to 

plan,  develop,  maintain,  deliver,  analyze  and 
validate  instruments  used  for  evaluation  purposes. 
Such  instruments  shall  include  behavioral 
objectives,  test  items  and  tests. 


DATA  INPUT:  AFS  Task  Data 

Publications 

Publication  Change  Notices 
Evaluation  Materials  Requirements 
Operational  Environment  Info 
Operational  Feedback 
Graphics 


DATA  OUTPUT:  BO  IDs 
BO  Data 

BO  &  Test  Data  for  Events  Scheduling 
Test  Item  IDs 
Test  Item  Data 
Test  Data 

Tmg/Evaluator  Resource  Data  for  Inventory 


STRUCTURE:  This  process  will  include  the  following  subprocesses 

-  Process  2.1.1,  Maintain  Behavioral  Objectives  Data 

-  Process  2.1.2,  Plan  Evaluation  Materials 

-  Process  2.1.3,  Maintain  Evaluation  Materials 
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PROCESS:  2.1.1,  Maintain  Behavioral  Objectives  (BO)  Data 


LEVEL  OF  AUTOMATION:  Automated,  automatic 


DESCRIPTION:  This  process  shall  enable  BO  developers/revisers  to 
develop,  revise,  save,  delete,  copy,  order,  review 
and  print  behavioral  objective  data  for  tasks  and 
subtasks.  This  process  shall  also  enable 
BO  developers/ revisers  to  maintain  a  Bad  Verb  List 
which  contains  unacceptable  verbs  that  are  matched 
against  the  verb  in  each  behavorial  objective 
statement  under  development  or  revision. 

Behavioral  objective  data  shall  be  maintained  for 
each  task  and  subtask.  BO  data  shall  be  keyed,  to 
make  evident  the  association  of  data  for  a  specific 
behavioral  objective  and  to  link  the  behavioral 
objective  to  the  respective  subtask  and/or  task. 


DATA  INPUT:  Evaluation  Materials  Requirements 

AFS  Task  Data 
Publications 

Publication  Change  Notice 


DATA  OUTPUT:  BO  IDs 
BO  Data 

Trng/Evaluator  Resource  Data  for  Inventory 
BO  Data  for  Event  Scheduling 

STRUCTURE:  This  process  will  include  the  following  subprocesses: 

2. 1.1.1,  Develop  BO  Data 

2. 1.1. 2,  Revise  BO  Data 

2. 1.1. 3,  Maintain  Bad  Verb  List 

Note:  No  futher  details  are  provided  within  this 
specification  for  the  functions  of  saving, 
deleting,  copying,  ordering,  reviewing  and 
printing  behavioral  objective  data. 
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Page  1  of  2 


PROCESS:  2. 1.1.1,  Develop  BO  Data 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  BO  developer  to  enter 

and/or  select  data  pertaining  to  a  specific 
behavioral  objective.  The  BO  developer  shall  be 
able  to: 

1.  Develop  the  behavioral  objective  statement, 

2.  Identify  references  (publications  and  their 
breakdowns)  that  apply  to  the  behavorial 
objective, 

3.  Identify  training,  evaluator  and  performance 
resources  that  apply  to  the  behavioral 
objective,  and 

4.  Identify  training  materials  that  apply  to  the 
behavioral  objective. 

This  process  shall  enable  a  person  to  develop 
BO  data  for  a  task  (called  a  terminal  behavioral 
objective)  or  for  a  subtask  (called  a  supporting 
behavioral  objective) .  Up  to  six  terminal 
objectives  shall  be  allowed  for  each  task  and 
up  to  six  supporting  objectives  shall  be  allowed 
for  each  subtask. 

The  developer  shall  review  task  analysis  data 
(task/ subtask  statement,  resources,  references, 
skills,  knowledges,  activities,  etc.)  to  determine 
the  BO  statement,  BO  references,  BO  resources  and 
BO  training  materials  that  apply  to  a  given 
task/subtask. 
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PROCESS : 

DESCRIPTION 


DATA  INPUT: 

DATA  OUTPUT 

STRUCTURE: 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Develop  BO  Data  (continued) 


(continued) 

A  Behavioral  Objective  ID  shall  be  automatically 
established  for  the  newly  developed  BO,  to  link  the 
BO  to  the  respective  task/ subtask  and  to  link  the 
BO  data  to  the  respective  behavioral  objective. 

To  develop  data  for  a  behavioral  objective, 
the  developer  shall  be  required  to  enter  or 
select  the  respective  Task/Subtask  ID  for 
which  the  behavioral  objective  applies. 

-  All  data  developed  for  an  objective  shall  be 
automatically  keyed  (linked)  to  the  behavioral 
objective  ID. 


AFS  Task  Data 
Bad  Verbs 
Publications 

BO  ID 
BO  Data 

Tmg/Evaluator  Resource  Data  for  Inventory 
BO  Data  for  Event  Scheduling 


This  process  includes  the  following  subprocesses: 


Process 

2. 1.1. 1.1 

View  Task/Subtask  Statement 

Process 

2. 1.1. 1.2 

Develop  BO  Statement 

Process 

2. 1.1. 1.3 

Identify  BO  References 

Process 

2. 1.1. 1.4 

Identify  BO  Resources 

Process 

2. 1.1. 1.5 

Specify  BO  Training  Materials 
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PROCESS:  2. 1.1. 1.1,  View  Task/ Subtask  Statement 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  BO  developer  to  review 

the  task  or  sub task  statement  which  applies  to  a 
given  behavioral  objective  under  development. 

Upon  the  developer's  request,  the  system  shall 
display  the  respective  task/ subtask  statement. 

The  task/subtask  statement  shall  not  be  altered  via 
this  process. 


DATA  INPUT:  Task/Subtask  Statement 


DATA  OUTPUT:  N/A 


STRUCTURE :  N/A 
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PROCESS:  2. l.l. 1.2,  Develop  BO  Statement 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  BO  developer  to  input 

a  narrative  behavioral  objective  statement  for  a 
given  task  or  subtask.  A  complete  behavioral 
objective  statement  shall  consist  of: 

1.  An  explicitly  stated  behavior, 

2.  •  The  operational  conditions  and  parameters, 

and 

3 .  The  standards  or  criteria  used  to  demonstrate 
trainee  mastery. 

This  process  shall  enable  the  developer  to  input 
the  behavioral  objective  statement  in  one  of  two 
ways: 

1.  The  EXPERT  mode  shall  enable  the  developer  to 
input  the  complete  behavioral  objective 
statement  in  one  continuous  step. 

2.  The  NOVICE  mode  shall  enable  the  developer  to 
separately  input  the  components  (behavior, 
conditions  and  standards)  of  the  behavioral 
objective  statement. 
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Develop  BO  Statement  (continued) 


(continued) 

In  the  EXPERT  mode,  the  developer  shall  be  able. to 
copy  and  accept  or  expand  the  task/subtask 
statement  to  develop  the  complete  BO  statement,. 

Once  the  complete  BO  statement  has  been  entered, 
the  statement  verb  shall  be  automatically  matched 
against  the  verbs  contained  on  the  Bad  Verb  List 
(see  process  2. 1.1. 3). 

If  the  statement  verb  matches  a  verb  on  the 
list,  the  behavioral  objective  statement  shall 
not  be  accepted  and  the  developer  shall  be 
required  to  re-enter  an  acceptable  verb. 

If  the  statement  verb  does  not  match  any  verb 
on  the  list,  the  BO  statement  shall  be 
accepted . 

In  the  NOVICE  mode,  the  developer  shall  be  able  to 
copy  and  accept  or  expand  the  task/subtask 
statement  to  develop  the  behavior  component  of  the 
BO  statement.  Once  the  behavior  component  has  been 
entered,  the  verb  shall  be  matched  against  the 
verbs  contained  on  the  Bad  Verb  List. 

-  If  the  behavior  component  verb  matches  a  verb 
on  the  list,  the  behavioral  component  shall 
not  be  accepted  and  the  developer  shall  be 
required  to  re-enter  an  acceptable  verb. 

If  the  behavior  component  verb  does  not  match 
any  of  the  verbs  on  the  list,  the  behavior 
component  shall  be  accepted. 
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STRUCTURE : 
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Develop  BO  statement  (continued) 


(continued) 

After  the  behavior  component  has  been  accepted, 
and  the  conditions  and  standards  components  have 
been  entered  by  the  developer,  this  process  shall 
automatically  combine  the  components  to  create  the 
complete  behavioral  objective  statement.  Then, 
the  developer  shall  be  able  to  modify  the  complete 
behavioral  objective  statement. 

Once  a  complete  BO  statement  has  been  accepted, 
in  either  the  EXPERT  or  NOVICE  mode,  a  unique 
Behavioral  Objective  ID  shall  be  automatically 
established.  The  objective  ID  shall  enable  the 
association  of  specific  objective  data  for  the 
BO,  as  well  as  the  association  of  the  BO  to  the 
respective  subtask/ task. 


Task/Subtask  Statement 
Bad  Verbs 
Publications 

Complete  BO  Statement 
BO  ID 


N/A 
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PROCESS:  2. 1.1. 1.3,  Identify  BO  References  (Publications) 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  BO  developer  to 

identify  the  task  performance  and  proficiency 
references  (publications)  which  pertain  to  a  given 
behavioral  objective.  This  process  involves  two 
functions: 

1.  Identifying  the  task/ subtask  publications 
which  apply  to  a  given  BO,  and 

2.  Specifying  the  publication  breakdowns  which 
apply  to  a  given  BO. 

The  developer  shall  identify  the  BO  references  by 
selecting  one  or  more  of  the  references  that  have 
been  defined  for  the  respective  task/ subtask.  At 
the  developer's  request,  the  system  shall  display 
the  task/ subtask  references.  The  developer  shall 
be  able  to  select  and  de~select  one  or  more  of  the 
displayed  references.  Selected  references  shall  be 
automatically  highlighted  or  identified  in  some 
other  way.  This  process  shall  not  permit  a 
developer  to  add  a  BO  reference  that  has  not 
already  been  identified  as  a  reference  for  the 
respective  task/ subtask. 

Once  the  behavioral  objective  publications  have 
been  identified,  the  developer  shall  be  able  to 
list  the  breakdowns  (e.g.,  paragraphs,  chapters, 
tables)  which  apply  to  any  of  the  BO  publications. 
Each  breakdown  shall  be  entered  by  keyboard  input. 
Developers  shall  be  able  to  add  or  delete 
breakdowns  to  or  from  the  list. 


DATA  INPUT:  Task/Subtask  References 

Publications 

DATA  OUTPUT:  BO  References 


STRUCTURE :  N/A 
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PROCESS:  2. 1.1. 1.4,  Identify  BO  Resources 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  BO  developer  to 

identify  the  performance,  training  and  evaluator 
resources  required  to  perform,  train  and  evaluate  a 
given  behavioral  objective.  This  process  involves 
three  functions: 

1.  Identifying  performance  resources,  which 

are  the  tools,  equipment  and  materials  (TEMs) 
necessary  to  accomplish  the  objective. 

2.  Identifying  the  training  resources  (e.g., 
slide  projector,  chalkboard)  necessary  when 
training  someone  to  accomplish  the  objective. 

3.  Identifying  the  evaluator  resources  (e.g., 
stopwatch,  clipboard)  necessary  when 
evaluating  someone  accomplishing  the 
objective. 

The  developer  shall  identify  the  BO  performance 
resources  by  selecting  one  or  more  of  the 
respective  task/ subtask  performance  resources. 

At  the  developer's  request,  the  system  shall 
automatically  display  the  task/ subtask  performance 
resources.  The  developer  shall  be  able  to  select 
and  de-select  one  or  more  of  the  performance 
resources  displayed.  Selected  resources  shall  be 
automatically  highlighted  or  identified  in  some 
other  way.  This  process  shall  not  permit  a 
developer  to  add  a  BO  performance  resource  that  has 
not  already  been  identified  as  a  performance 
resource  for  the  respective  task/subtask. 
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STRUCTURE: 
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Identify  BO  Resources  (continued) 


(continued) 

The  developer  shall  be  able  to  list  the  BO  training 
and  evaluator  resources,  by  entering  the  name  of 
each  resource.  Training  resources  shall  be  listed 
separately  from  evaluator  resources.  The  developer 
shall  be  able  to  add  or  delete  training  or 
evaluator  resources  to  or  from  the  respective  list. 

If  a  training/evaluator  resource  is  not  one  of 
the  resources  for  which  data  are  already  being 
maintained  by  the  system,  the  BO  developer 
shall  be  able  to  enter  additional  data  to  add 
the  resource  to  the  system  (process  1.3.1). 


Task/Subtask  Performance  Resources 


BO  Resources 

Tmg/Evaluator  Resource  Data  for  Inventory 
N/A 
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PROCESS:  2. 1.1. 1.5,  Specify  BO  Training  Materials 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  BO  developer  to  list 

the  training  materials  that  are  associated  with  a 
given  behavioral  objective.  Training  materials 
shall  include  Computer  Assisted  Instruction  (CAI) 
lessons,  textbooks,  training  handbooks.  Interactive 
Video  Disk  (IVD)  lessons,  films,  and  sound-on-slide 
presentations ,  among  others . 

Training  Materials  shall  be  identified  by  type  of 
material  (e.g,  CAI,  text,  film),  ID  and 
name/description;  these  data  shall  be  entered  by 
keyboard  input.  Developers  shall  be  able  to  add 
or  delete  materials  to  or  from  the  list. 


DATA  INPUT:  N/A 


DATA  OUTPUT:  BO  Training  Materials 


STRUCTURE:  N/A 
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PROCESS:  2. 1.1. 2,  Revise  BO  Data 


LEVEL  OF  AUTOMATION;  Automated 


DESCRIPTION:  This  process  shall  enable  a  BO  reviser  to:  revise  a 

behavioral  objective  statement  and  any  references, 
resources  and  training  materials  that  apply  to  a 
specific  behavioral  objective. 

Data  for  a  behavioral  objective  may  require 
revision  because  of  task  performance  or  proficiency 
changes  which  occur  to  publications  governing 
the  task/subtask.  Such  changes  in  publications 
shall  be  identified  (flagged)  by  the  Management 
Subsystem,  to  alert  a  reviser  that  he/she  should 
review  the  publications  to  determine  if  the 
respective  BO  data  require  changing.  Changes  in 
task/subtask  publications  may  also  cause  changes 
in  the  task/ subtask  statement,  resources, 
references,  skills  and  knowledges  and/or 
activities.  The  reviser  shall  be  able  to  display 
or  print  these  task  data  before  revisions  to  BO 
data  occurs. 

The  reviser  shall  be  able  to  review  the  respective 
task/ subtask  statement  (same  as  Process  2.1. 1.1.1, 
View  Task/Subtask  Statement) . 
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DATA  OUTPUT: 

STRUCTURE : 
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Revise  BO  Data  (continued) 


(continued) 

When  a  behavioral  objective  statement  requires 
revision,  the  revisor  shall  be  able  to  modify  any 
part  or  all  of  the  complete  BO  statement.  Once  the 
revised  BO  statement  has  been  entered,  the 
statement  verb  shall  be  automatically  matched 
against  the  verbs  contained  on  the  Bad  Verb  List. 

If  the  statement  verb  matches  a  verb  on  the 
Bad  Verb  List,  the  revised  BO  statement  shall 
not  be  accepted  and  the  reviser  shall  be 
required  to  re-enter  an  acceptable  verb. 

If  the  statement  verb  does  not  match  a  verb  on 
the  Bad  Verb  List,  the  revised  BO  statement 
shall  be  accepted. 

When  BO  references  (publications)  require  revision, 
the  same  functions  that  apply  to  Process  2. 1.1. 1.3, 
Identify  BO  References,  shall  apply  for  this 
process . 

When  BO  resources  require  revision,  the  same 
functions  that  apply  to  Process  2. 1.1. 1.4,  Identify 
BO  Resources,  shall  apply  for  this  process. 

When  BO  training  materials  require  revision,  the 
same  functions  that  apply  to  Process  2 . 1 . 1 . 1 . 5 , 
Specify  BO  Training  Materials,  shall  apply  for  this 
process. 

BO  Data 

AFS  Task  Data 

Publications 

Publication  Change  Notice 
Bad  Verbs 


Revised  BO  Data 

Tnrg/Evaluator  Resource  Data  for  Inventory 
N/A 
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PROCESS:  2. 1.1. 3,  Maintain  Bad  Verb  List 


LEVEL  OF  AUTOMATION:  Automated,  automatic 


DESCRIPTION:  This  process  shall  enable  BO  developers/revisers  to 
maintain  an  on-line  list  of  unacceptable  BO 
statement  verbs.  Developers/revisers  shall  be  able 
to:  add  verbs  to  the  list;  delete  verbs  from  the 
list;  and  review  the  list  on  line. 

The  system  shall  automatically  call  up  the 
Bad  Verb  List  each  time  a  person  develops  or 
revises  a  behavioral  objective  statement.  The  verb 
contained  in  a  newly  developed  or  revised 
behavioral  objective  statement  shall  be  matched 
against  each  verb  contained  on  the  Bad  Verb  List. 

If  a  BO  statement  verb  matches  a  verb  on  the 
Bad  Verb  List,  the  BO  statement  shall  be  rejected. 


DATA  INPUT:  Bad  Verbs 


DATA  OUTPUT:  Bad  Verbs 


STRUCTURE:  N/A 
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PROCESS :  2.1.2.,  Plan  Evaluation  Materials 


LEVEL  OF  AUTOMATION:  Manual,  Automated 


DESCRIPTION:  This  process  shall  enable  individuals  to  plan 

evaluation  materials  for  one  or  more  tasks  by: 

1.  Obtaining,  reviewing  and  analyzing  task 
and  behavioral  objective  information,  and 

2 .  Determining  appropriate  evaluation 
strategies  and  resource  requirements. 

Some  of  the  task  data  required  to  accomplish  this 
process  shall  be  printed  or  reviewed  on  line. 

Other  task-related  data  shall  be  obtained  by 
reviewing  task  performance  and  proficiency  data 
(publications) ,  observing  the  workcenter 
environment  (available  resources,  noise  control, 
facility  accommodations,  etc.)  and  by  interviewing 
workcenter  personnel. 

By  considering  evaluation  parameters  (task  skill 
and  knowledge  requirements,  task  performance 
literacy  requirements,  etc.)  and  weighing  factors 
(cost,  time,  fidelity  and  resource  availability) , 
an  evaluation  materials  planner  shall  be  able  to 
determine  the  appropriate  evaluation  strategies 
(over-the-shoulder  observation,  knowledge  testing, 
product  evaluation,  etc.)  and  resource  requirements 
for  evaluating  persons  on  a  given  task. 

DATA  INPUT:  Operational  Environment  Info 

AFS  Task  Data 
Publications 
BO  Data 

DATA  OUTPUT:  Task  Evaluation  Strategy  Info 


STRUCTURE:  This  process  includes  the  following  subprocesses: 

-  Process  2. 1.2.1,  Review/ Analyze  Task  Info 

-  Process  2. 1.2. 2,  Determine  Evaluation  Strategies  & 

Resource  Requirements. 
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PROCESS:  2. 1.2.1,  Review/Analyze  Task  Info 


LEVEL  OF  AUTOMATION:  Automated,  manual 


DESCRIPTION:  This  process  shall  enable  an  evaluation  materials 

planner  to  obtain  information  pertaining  to  a 
specific  task,  from  which  evaluation  parameters 
will  be  derived.  The  evaluation  parameters  shall 
be  used  to  determine  evaluation  strategies 
appropriate  for  the  task. 

The  evaluation  materials  planner  shall  obtain 
specific  task  information  by: 

1.  Reviewing  task  analysis  data  (references, 
activities,  performance  resources,  weapon 
systems,  etc.),  either  on  line  or  once 
printed  in  hardcopy. 

2.  Reviewing  behavioral  objectives  data,  either 
on  line  or  once  printed  in  hardcopy. 

3.  Reviewing  sources  of  task  performance  and 
proficiency  (publications) . 

4.  Observing  the  operational  environment  in 
which  the  task  is  performed. 

5.  Interviewing  operational  workcenter 
personnel . 
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DATA  INPUT: 

DATA  OUTPUT: 

STRUCTURE: 
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Review/ Analyze  Task  Info  (continued) 


(continued) 

The  collected  information  shall  be  analyzed  to 
determine  the  evaluation  parameters.  Evaluation 
parameters  shall  include: 

1.  Task  skill  and  knowledge  requirements  which 
must  be  evaluated, 

2.  Task  performance  literacy  requirements 
(nomenclature)  which  must  be  evaluated, 

3.  Influencing  environmental  conditions  which 
must  be  considered, 

4.  Task  performance  variance  across  work  centers 
which  must  be  estimated,  and 

5.  Probability  of  change  within  work  centers. 

Once  the  evaluation  parameters  have  been 
established,  the  evaluation  planner  shall  use  the 
information  to  determine  the  evaluation  strategies 
most  approriate  for  the  task  and  the  resources 
which  are  required  to  accomplish  the  evaluation (s) . 


AFS  Task  Data 

Operational  Environment  Info 

Publications 

BO  Data 


Task  Evaluation  Parameters 


N/A 
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PROCESS:  2. 1.2. 2,  Determine  Evaluation  Strategies  & 

Resource  Requirements 


LEVEL  OF  AUTOMATION:  Manual 


DESCRIPTION:  This  process  shall  enable  an  evaluation  materials 
planner  to  assess  evaluation  parameters  and 
weighted  factors  applicable  to  a  specific  task  in 
order  to  determine  evaluation  strategies  most 
appropriate  for  the  task. 

The  evaluation  materials  planner  will  assess  the 
evaluation  parameters  derived  from  Process  2. 1.2.1, 
Review/ Analyze  Task  Info.  The  planner  will  also 
weigh  factors  such  as  cost,  time,  fidelity, 
resource  availability  and  priority  of  the  task. 

The  analysis  of  the  parameters  and  weighted  factors 
shall  enable  the  planner  to: 

1.  Select  the  evaluation  strategies  that  are 
acceptable  for  evaluating  the  task, 

2.  Establish  the  resource  requirements  that 
apply  to  each  selected  evaluation  stategy, 

3 .  Compare  the  resource  requirements  to 
resource  availability, 

4.  Rank  the  selected  strategies, 

5.  Establish  resource  requirements  for 

the  evaluation  strategies  which  shall  be 
used  to  develop  the  evaluation  materials. 
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PROCESS : 

DESCRIPTION: 


DATA  INPUT: 

DATA  OUTPUT: 

STRUCTURE : 


Determine  Evaluation  Strategies  &  Resource 
Requirements  (continued) 


(continued) 

The  evaluation  strategies  determined  from  this 
process  shall  support  the  direct  observation  of 
airmen  task  performance  for  certification.  In 
instances  where  direct  observation  of  a  task 
is  difficult  or  cannot  be  performed,  alternates 
to  direct  observation  shall  be  specified. 

Adequate  strategies  with  low  resource  requirements 
shall  be  given  priority.  Evaluation  strategies 
shall  include: 

1.  Over-the-shoulder  observation, 

2.  Knowledge  testing  (true/false,  multiple 
choice,  matching,  etc.), 

3.  Product  evaluation, 

4.  Rigged  (scenario), 

5.  Simulation, 

6.  other. 


Task  Evaluation  Parameters 


Task  Evaluation  Strategy  Info 
N/A 
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PROCESS:  2.1.3,  Maintain  Evaluation  Materials 


LEVEL  OF  AUTOMATION:  Manual,  Automated,  Automatic 


DESCRIPTION:  This  process  shall  enable  evaluation  materials 

developers  to  develop,  revise,  save,  delete, 
review,  print,  analyze  and  validate  evaluation 
materials.  Evaluation  materials  include  test  items 
and  tests . 


DATA  INPUT:  AFS  Task  Data 

Publications 

Publication  Change  Notice 
BO  Data 

Task  Eval  Strategy  Info 
Operational  Feedback 
Graphic (s) 

Test  Score  Data 


DATA  OUTPUT:  .Test  IDs 

Test  Item  IDs 

Test  Data 

Test  Item  Data 

Test  Item  Analysis  Keys 

Graphic (s)  ID(s) 

STRUCTURE:  This  process  includes  the  following  subprocesses : 

-  Process  2.1.1,  Maintain  Test  Item  Bank 

-  Process  2.1.2,  Maintain  Tests 


72 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 

PROCESS:  2. 1.3.1,  Maintain  Test  Item  Bank 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  test  item  developers  and 

revisers  to  develop,  list,  revise,  save,  copy, 
delete,  review  and  print  test  items?  and  thereby 
maintain  the  Test  Item  Bank.  The  Test  Item  Bank 
shall  act  as  the  central  depository  for  all 
test  item  data. 

For  each  test  item,  this  process  shall  enable 
developing/revising  of  on-line  text,  specifying 
the  test  item  references  which  apply,  and  coding 
the  item  to  the  respective  knowledge  or  performance 
test(s)  once  one  or  more  tests  have  been 
developed  which  incorporate  the  test  item. 


DATA  INPUT:  AFS  Task  Data 

Publications 
BO  Data 
Graphics 

Task  Evaluation  Strategy  Info 
Test  ID 

Test  Item  Revision  Requirement (s) 
Publication  Change  Notice 


DATA  OUTPUT:  Test  Item  IDs 
Test  Item  Data 


STRUCTURE:  This  process  includes  the  following  subprocesses: 

Process  2. 1.3.1,  Develop/Revise  Test  Item 

-  Process  2. 1.3. 2,  Specify  Test  Item  References 

Process  2. 1.3. 3,  Code  Test  Item  to  Test 

The  functions  of  list  all  test  items  for  an 
objective,  save  an  item,  copy  an  item,  delete  an 
item,  review  an  item  and  print  an  item  are  not 
covered  in  any  further  detail  within  this 
specification. 
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PROCESS:  2. 1.3. 1.1,  Develop/Revise  Test  Item 

LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  item  developer  or 

reviser  to  develop  or  revise  a  specific  test  item. 

A  test  item  shall  be  one  of  the  following: 

1.  An  Oral  Test  Guide  (OTG) ,  which  contains 
instructions  to  the  evaluator  for 
administering  a  performance  evaluation. 

2.  A  Performance  Evaluation  Checklist  (PEC), 
which  is  a  checklist  to  be  used  by  the 
evaluator  when  conducting  a  performance 
evalution. 

3.  A  knowledge  test  item,  which  is  a  question 
to  be  part  of  a  knowledge  test.  A 
knowledge  test  item  shall  be  one  of  the 
following  type  questions: 

a.  True/False 

b.  Multiple  Choice 

c.  List  Multiple  Choice 

d.  Constructed  respone  (f ill-in-the-blank) 

e .  Touch 

f.  Touch  Sequence 

g.  Matching 

Based  on  task  evaluation  strategies  and  resource 
determinations  for  a  given  task  (refer  to  Process 
2. 1.2. 2),  the  test  item  developer/reviser  shall  be 
able  to  develop  and  revise  one  or  more  specific 
test  items  which  will  be  incorported  into  one  or 
more  tests  to  evaluate  trainee's  knowledge  and/or 
performance  of  the  task. 
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PROCESS : 

DESCRIPTION: 


Develop/Revise  Test  Item  (continued) 


(continued) 

Task  analysis  and  behavioral  objective  data  shall 
be  used  when  developing  or  revising  a  test  item. 
Test  Item  data  shall  be  keyed,  to  make  evident 
the  association  of  data  for  a  specific  test  item 
and  to  link  the  test  item  to  the  respective 
behavioral  objective.  Once  a  test  item  has  been 
developed  and  saved  it  shall  be  assigned  a  Test 
Item  Id;  and  added  to  the  Test  Item  Bank,  which  is 
the  central  depository  for  all  data  for  all  test 
items  developed  using  the  AOTS.  Based  on  changes 
in  task  performance  and  proficiency  sources 
(publications) ,  evaluation  analysis  which 
determined  an  item  to  be  inadequate,  or  for  other 
required  reasons,  a  test  item  reviser  shall  be  able 
to  revise  any  portion  or  all  of  the  data  for  a  test 
item. 


To  develop  data  for  a  new  test  item,  the 
developer  shall  be  able  to  either  enter  the 
complete  test  item  data  or  copy  another 
test  item  and  modify  the  copied  item's  data. 

To  develop  data  for  a  new  test  item,  the 
developer  shall  be  required  to  enter  or 
select  the  respective  Task  ID,  Subtask  ID 
(if  applicable)  and  Behavioral  Objective  ID 
for  which  the  test  item  is  being  developed. 

To  access  an  existing  test  item,  the  reviser 
shall  be  required,  as  a  minimum,  to  enter  or 
select  the  Test  Item  ID. 

Development/ revision  documentation  (e.g., 
test  item  author,  modification  date)  shall  be 
automatically  maintained  by  the  system. 
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PROCESS:  Develop/Revise  Test  Item  (continued) 


DATA  INPUT:  AFS  Task  Data 

BO  Data 
Publications 

Task  Evaluation  Strategy  Info 
Graphics 

Publication  Change  Notice 

Test  Item  Revision  Requirement (s) 

Test  Item 


DATA  OUTPUT:  Test  Item  ID 
Test  Item 


STRUCTURE:  This  process  includes  the  following  subprocesses : 

Process  2 . 1 . 3 . 1 . 1 ,  Develop/Revise  OTG 

-  Process  2. 1.3. 1.2,  Develop/Revise  PEC 

-  Process  2. 1.3. 1.3,  Develop/Revise  Knowledge 

Test  Item 
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PROCESS:  2. 1.3. 1.1.1,  Develop/Revise  OTG 

LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  item  developer  or 

reviser  to  develop  or  revise  on-line  data  for  an 
Oral  Test  Guide  (OTG) .  An  OTG  shall  contain 
instructions  to  the  evaluator  for  administering 
a  performance  or  product  evaluation  for  a  specific 
task  or  subtask.  The  OTG  shall  later  be  coupled 
with  a  Performance  Evaluation  Checklist  (PEC)  to 
develop  a  performance  test  (see  Process  2. 1.3. 2.1, 
Develop/Revise  Test) .  This  process  shall  also 
automatically  maintain  test  item  development  data. 

The  test  item  developer  or  reviser  shall  enter  text 
for  the  OTG  by  keyboard  input.  A  developer  shall 
be  able  to  copy  another  OTG  and  modify  the  copied 
OTG  contents  in  order  to  create  a  new  OTG.  The 
system  shall  permit  up  to  5  screens  (20  lines  per 
screen)  of  data  for  each  OTG.  This  process  shall 
enable  the  test  item  developer/ reviser  to  add, 
revise  or  delete  data  for  any  OTG  for  which  he/she 
is  authorized  access.  OTG  data  shall  include: 

1.  Evaluator  information  (estimated  time  to 
accomplish  evaluation;  tools,  equipment 
and  materials  required  to  accomplish 
evaluation,  etc.),  and 

2.  Specific  evaluator  instructions  (how  to 
administer  the  performance  evaluation,  what 
to  say  to  the  trainee,  etc.). 

A  Test  Item  ID  shall  be  automatically  assigned  once 
the  OTG  text  has  been  developed  and  saved.  OTG 
development  and  revision  documentation  (author, 
modification  date,  etc.)  shall  be  automatically 
maintained  by  the  system. 
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DESCRIPTION: 


DATA  INPUT: 

DATA  OUTPUT: 

STRUCTURE: 


Specification  Number  70S647300B 
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Develop/Revise  OTG  (continued) 


(continued) 

Evaluation  strategies  and  resource  availability 
determinations,  task  analysis  data  and  behavioral 
objective  data  shall  be  reviewed  and  used  when 
developing  or  revising  an  OTG. 

Based  on  changes  in  task  performance  and 
proficiency  sources  (publications) ,  evaluation 
analysis  results  which  identify  an  OTG  to  be 
inadequate,  or  for  other  required  reasons,  a  test 
item  reviser  shall  be  able  to  revise  any  portion 
or  all  of  the  data  for  the  OTG. 


AFS  Task  Data 
BO  Data 
Publications 

Task  Evaluation  Strategy  Info 
Publication  Change  Notice 
Test  Item  Revision  Requirement (s) 
OTG 


OTG 

Test  Item  ID 


N/A 
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-PROCESS:  2. 1.3. 1.1. 2,  Develop/Revise  PEC 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  item  developer  or 

reviser  to  develop  or  revise  on-line  data  for  a 
Performance  Evaluation  Checklist  (PEC) .  The  PEC  is 
a  checklist  to  be  used  by  an  evaluator  when 
conducting  a  performance  or  product  evaluation  for 
a  specific  task/ subtask.  The  PEC  shall  contain  the 
steps  which  are  performed  by  the  trainee  and 
observed  by  the  evaluator.  The  PEC  shall  later  be 
coupled  with  an  Oral  Test  Guide  (OTG)  to  create  a 
performance  test  (see  process  2. 1.3. 2.1, 
Develop/Revise  Test) .  This  process  shall  also 
automatically  maintain  test  item  development  data. 

A  template  shall  be  developed  and  automated  which 
shall  enable  a  PEC  developer  or  reviser  to  enter 
text  for  each  step  of  the  PEC.  The  template  shall 
support  up  to  60  performance  steps  for  a  PEC.  At 
the  developer's/ reviser's  request,  the  template 
shall  be  called  up  and  the  PEC  developer  or  reviser 
shall  be  able  to  add,  delete,  or  revise  one  or  more 
steps.  The  steps  of  the  PEC  shall  be  automatically 
sequenced  in  numerical  order,  as  steps  are  added 
and  deleted  from  the  PEC;  upon  request,  the 
developer  or  reviser  shall  be  able  to  reorder  the 
PEC  steps.  When  a  performance  test  is  printed 
which  involves  a  PEC,  space  shall  be  provided  and 
identified,  beside  each  PEC  step,  for  the  observer 
to  record  his/her  responses  (yes,  no,  unobserved) . 

This  process  shall  enable  the  developer/reviser 
to  designate  one  or  more  PEC  steps  as  critical. 
Critical  steps  not  successfully  performed  shall 
cause  a  trainee  to  fail  the  evaluation.  Designated 
critical  steps  shall  be  highlighted  or  identified 
in  some  other  way. 


79 


Page  2  of  2 
PROCESS : 

DESCRIPTION: 


DATA  INPUT: 

DATA  OUTPUT: 

STRUCTURE: 
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Develop/Revise  PEC  (continued) 


(continued) 

A  Test  Item  ID  shall  automatically  be  assigned  once 
a  PEC  is  developed  and  saved.  PEC  development 
and  revision  documentation  (author,  modification 
date,  etc)  shall  be  automatically  maintained  by 
the  system. 

Evaluation  strategies  and  resource  availability 
determinations,  task  analysis  data  and  behavioral 
objective  data  shall  be  reviewed  and  used  when 
developing  or  revising  a  PEC. 

Based  on  changes  in  task  performance  and 
proficiency  sources  (publications) ,  evaluation 
analysis  results  which  identify  a  PEC  to  be 
inadequate,  or  for  other  required  reasons,  a 
test  item  reviser  shall  be  able  to  revise  any 
portion  or  all  of  the  data  for  the  PEC. 


AFS  Task  Data 
BO  Data 
Publications 

Task  Evaluation  Stragegy  Info 
Publication  Change  Notice 
Test  Item  Revision  Requirement (s) 
PEC 


PEC 

Test  Item  ID 


N/A 
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PROCESS :  2. 1.3. 1.1. 3,  Develop/Revise  Knowledge  Test  Item 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  item  developer  or 
reviser  to  develop  or  revise  on-line  data  for  a 
knowledge  test  item.  This  process  shall 
automatically  maintain  test  item  development  data 
(author,  date  last  modified,  etc.)  for  the  item  and 
enable  the  developer/ reviser  to  playback  the  item. 

A  knowledge  test  item  shall  be  incorporated  into 
one  or  more  knowledge  tests.  Because  a  knowledge 
test  may  be  administered  on  line  or  off  line,  this 
process  shall  enable  the  developer/revisor  to  input 
data  which  supports  on-line  or  off-line 
presentation  of  the  test  item. 

A  test  item  developer  shall  be  able  to  develop 
a  new  knowledge  test  item  by  either  entering  all 
new  data  or  by  copying  an  existing  knowledge  test 
item  and  modifying  the  copied  item's  contents. 

A  knowledge  test  item  shall  be  one  of  the  following 
types : 


1.  True/False 

2 .  Multiple  Choice 

3.  List  Multiple  Choice 

4.  Constructed  Response  (fill-in-the-blank) 

5 .  Touch 

6.  Touch  sequence 

7 .  Matching 

The  test  item  developer/reviser  shall  be  able  to 
input  the  following  data  for  each  knowledge  test 
item: 


1 .  Item  stem 

2.  Graphics  (optional) 

3.  Item  alternative (s) 

4 .  Item  feedback 
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Develop/Revise  Knowledge  Test  Item  (continued) 


(continued)  ■ 

The  item  stem  shall  be  entered  by  keyboard 
input,  with  text  editing  support  provided  by  the 
system.  This  process  shall  permit  up  to  two 
screens  (20  lines  per  screen)  for  each  item  stem. 

-  Graphics  shall  be  permitted  for  test  items 

which  are  developed  for  on-line  use  only.  Each 
graphic  shall  shall  be  considered  as  part  of 
the  item  stem.  Up  to  three  graphics  shall  be 
permitted  for  a  given  test  item.  Once  the  test 
item  developer  provides  the  valid  ID/name  for  a 
specific  graphic,  the  graphic  shall  be 
automatically  displayed.  -The  test  item 
developer/ reviser  shall  be  able  to  position, 
rotate,  scale,  and  color  the  displayed  graphic. 

Capabilities  shall  be  provided  to  enable  the  test 
item  developer/revisor  to  input  item  alternatives. 
These  capabilities  shall  also  enable  the 
developer/reviser  to  flag  (code)  the  correct 
alternative ( s) ,  so  scoring  and  analysis  for  the 
item  can  occur. 

Capabilities  to  provide  feedback  shall  be  permitted 
for  all  types  of  knowledge  test  items. 

Evaluation  stategies  and  resource  availability 
determinations,  task  analysis  data  and  behavioral 
objective  data  shall  be  reviewed  and  used  when 
developing  or  revising  a  knowledge  test  item. 

A  Test  Item  ID  shall  be  automatically  assigned  once 
data  for  a  knowledge  test  item  has  been  developed 
and  saved.  Test  item  development  and  revision  data 
(author,  modification  date,  etc.)  shall  be 
automatically  maintained  by  the  system. 
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DATA  OUTPUT: 
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Develop/Revise  Knowledge  Test  Item  (continued) 


(continued) 

Based  on  changes  in  task  performance  and 
proficiency  sources  (publications) ,  evaluation 
analysis  results  which  identify  a  knowledge  test 
item  to  be  inadequate,  or  for  other  required 
reasons,  a  test  item  reviser  shall  be  able  to 
revise  any  portion  or  all  of  the  data  for  the 
item. 


AFS  Task  Data 
BO  Data 
Publications 

Task  Evaluation  Strategy  Info 
Graphic (s) 

Publication  Change  Notice 

Test  Item  Revision  Requirement (s) 

Knowledge  Test  Item 


Graphic (s)  ID(s) 
Knowledge  Test  Item 
Test  Item  ID 


STRUCTURE: 


N/A 
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PROCESS:  2. 1.3. 1.2,  Specify  Test  Item  References 

(Publications) 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  item  developer  or 

reviser  to  specify  the  task  performance  and 
proficiency  references  (publications)  which  pertain 
to  a  given  test  item.  This  process  involves  two 
functions: 

1.  Identifying  the  task/subtask  publications 
which  apply  to  the  test  item,  and 

2.  Specifying  the  publication  breakdowns  which 
apply  to  the  test  item. 

The  developer/reviser  shall  identify  the  test  item 
references  based  on  the  references  that  have  been 
defined  for  the  behavioral  objective  to  which  the 
test  item  applies.  Upon  the  user's  request,  the 
system  shall  automatically  display  the  objective 
references.  The  developer/reviser  shall  be  able 
to  select  and  de-select  one  or  more  of  the 
references  displayed.  Selected  references  shall  be 
automatically  highlighted  or  identified  in  some 
other  way.  This  process  shall  not  permit  a  test 
item  developer  or  reviser  to  add  a  reference  that 
has  not  already  been  identified  as  a  reference  for 
the  respective  behavioral  objective. 

Once  the  test  item  publications  have  been 
identified,  the  developer/ reviser  shall  be  able  to 
list  the  breakdowns  (paragraphs,  chapters,  tables, 
etc.)  which  apply  to  any  of  the  identified 
publications.  Each  breakdown  shall  be  entered  by 
keyboard  input.  Developers/ revisers  shall  be  able 
add  or  delete  breakdowns  to  or  from  the  list. 
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DATA  OUTPUT: 

STRUCTURE  r 
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Specify  Test  Item  References  (continued) 


(continued) 

Based  on  changes  in  task  performance  and 
proficiency  sources  (publications)  identified  by 
the  Management  Subsystem,  a  test  item  reviser  shall 
be  able  to  re-specify  any  one,  or  all,  of  the  test 
item  references  or  breakdowns. 


BO  References 
Publications 

Publication  change  Notice 
Test  Item  References 


Test  Item  References 
N/A 
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PROCESS:  2. 1.3. 1.3,  Code  Test  Item  To  Test 


LEVEL  OF  AUTOMATION:  Automatic 


DESCRIPTION:  This  process  shall  automatically  link  a  given  test 

item  (PEC,  OTG  or  knowledge  test  item)  to  the 
performance  or  knowledge  test(s)  which  incorporate 
the  test  item. 

Each  time  a  test  is  developed  or  revised,  the  Test 
ID  shall  be  automatically  linked  (coded)  to  each 
test  item  incorporated  within  the  test. 

Performance  Test  IDs  shall  be  coded  to  OTGs  and 
PECs;  while  knowledge  Test  IDs  shall  be  coded  to 
knowledge  test  items.  This  coding  shall  enable: 

1.  Correct  test  item  presentation  for  a  given 
performance  or  knowledge  test  at  the 

time  the  test  is  printed  for  off-line  use 
or  administered  on  line. 

2.  Evaluation  analysis  tp  be  performed  for 
a  given  test  item. 


DATA  INPUT:  Test  Id 


DATA  OUTPUT:  Test  Id 


STRUCTURE :  N/A 
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PROCESS:  2. 1.3. 2,  Maintain  Tests 

LEVEL  OF  AUTOMATION:  Automated,  Automatic,  Manual 


DESCRIPTION:  This  process  shall  enable  test  developers/revisers 

to  develop,  revise,  review,  print  and  delete  tests. 
Two  types  of  tests  shall  be  maintained: 

1.  Knowledge  tests,  which  shall  be  used  for 
measuring  trainee's  task/ subtask  knowledge, 
and 

2.  Performance  tests,  which  shall  be  used  for 
measuring  trainee's  task/subtask  performance 
or  end  products. 

Knowledge  tests  shall  be  developed  for  on-line 
or  off-line  administration;  and  shall  support  pre¬ 
training  and  post-training  evaluations. 

Performance  tests  shall  be  developed  for  off-line 
administration;  and  shall  support  pre-training, 
post-training  and  Quality  Control  evaluations. 

This  process  shall  also  enable  test  developers/ 
revisers  to  analyze  and  validate  evaluation 
materials. 


DATA  INPUT:  AFS  Task  Data 

Publications 
BO  Data 

Test  Item  Data 

Task  Evaluation  Strategy  Info 
Publication  Change  Notice 
Operational  Feedback 
Test  Score  Data 


DATA  OUTPUT:  Test  IDs 
Test  Data 

Test  Data  for  Event  Scheduling 
Test  Item  Revision  Requirement (s) 
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PROCESS:  Maintain  Tests  (continued) 


STRUCTURE:  This  process  includes  the  following  subprocesses : 

-  Process  2 . 1 . 3 . 2 . 1 ,  Develop/Revise  Test 

-  Process  2. 1.3. 2. 2,  Analyze  Evaluation  Materials 

-  Process  2. 1.3. 2. 3,  Validate  Evaluation . Materials 

NOTE:  No  further  details  are  provided  within 
this  specification  for  the  functions  of 
saving,  deleting,  reviewing  and  printing 
tests . 
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PROCESS:  2. 1.3. 2.1,  Develop/Revise  Test 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  developer  to 

develop  or  revise  a  test  on  line.  Capabilities 
shall  exist  to  permit  the  development  of  either 
a  knowledge  or  performance  test. 

-  A  knowledge  test  shall:  consist  of  one  or  more 
knowledge  test  items  (true/false,  multiple 
choice,  etc.);  be  developed  for  on-line  and/or 
off-line  administration;  and  support 
pre-training  or  post-training  evaluation  for 
an  AFS  task  or  subtask. 

-  A  performance  test  shall:  consist  of  one  or 
more  Oral  Test  Guides  (OTGs)  and  one  or  more 
Performance  Evaluation  Checklists  (PECs) ;  be 
developed  for  off-line  administration;  and 
support  pre-training  or  post-training 
evaluation  for  an  AFS  task  or  subtask,  or 
Quality  Control  evaluation. 

This  process  shall  involve  the  following: 

1.  Identifying  the  objectives  for  which  the 
test  is  being  developed  and  the  test  items 
to  be  incorporated  into  the  test, 

2.  Formatting  the  test, 

3.  Establishing  test  parameters, 

4.  Developing  on-line  and/or  off-line 
instructions  for  taking  the  test  (knowledge 
test  only) ,  and 

5.  Maintaining  test  modification  data. 
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PROCESS : 

DESCRIPTION: 


DATA  INPUT: 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Develop/Revise  Test  (continued) 


(continued) 

A  test  developer/reviser  shall  consider 
evaluation  strategies,  resource  availability, 
task  analysis  data  (activities,  skills  and 
knowledges,  etc.)  and  behavioral  objective  data 
when  developing  or  revising  a  test.  These  data 
will  help  in  identifying  the  type  of  test  to 
develop  and  the  most  appropriate  test  items  for  the 
test.  Each  test  shall  be  keyed  either  to  a 
terminal  or  supporting  objective.  When  a  new  test 
is  developed  a  test  number  shall  be  automatically 
assigned.  This  numbered  Test  ID  shall  be  used  to 
distinguish  the  test  from  other  tests  developed  for 
the  objective  and  to  associate  the  test  items  to 
the  test. 

Based  on  changes  in  task  performance  and 
proficiency  sources  (publications) ,  evaluation 
analysis  which  determine  a  test  to  be  inadequate, 
or  other  reasons,  a  test  reviser  shall  be  able  to 
revise  any  portion  or  all  of  the  data  for  a  test. 

Test  development  and  modification  data  (author, 
date  last  modified,  etc.)  shall  be  automatically 
maintained.  The  original  test  developer's  name  and 
initial  test  development  date  shall  be  maintained 
until  the  test  is  deleted.  Each  time 
objective/ test  item  selection,  test  format,  test 
parameters  or  test  instructions  data  are  revised 
for  a  test,  the  test  reviser's  name  and  the 
modification  date  shall  be  updated. 


AFS  Task  Data 
Publications 
BO  Data 
Test  Data 
Test  Item  Data 

Task  Evaluation  Strategy  Info 
Test  Revision  Requirement (s) 
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PROCESS : 

DATA  OUTPUT: 

STRUCTURE : 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Develop/Revise  Test  (continued) 


Test  ID 
Test  Data 

Test  Data  for  Event  Scheduling 


This  process  includes  the  following  suoprocesses: 

-  Process  2. 1.3. 2. 1.1,  Select  Objectives  and  Test 

Items 

-  Process  2. 1.3. 2. 1.2,  Input  Test  Format 

-  Process  2. 1.3. 2. 1.3,  Input  Test  Parameters 

-  Process  2. 1.3. 2. 1.4,  Input  Knowledge  Test 

Instructions 


91 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  1  of  4 


PROCESS:  2. 1.3. 2. 1.1,  Select  Objectives  and  Test  Items 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  developer  or 

reviser  to  select  the  behavioral  objective (s) 
and  test  items  for  a  test  being  developed 
or  revised. 

Each  test  shall  be  keyed  to  a  terminal  (task) 
or  a  supporting  (subtask)  objective.  Once  a  test 
has  been  keyed  to  a  particular  behavioral 
objective,  the  test  developer  shall  be  able  to: 

1.  Select  and  include  test  items  applicable  to 
the  keyed  objective,  and 

2.  Select  any  other  objective  existing  for 
the  task;  and  select  and  include  any  test 
items  which  apply  to  that  objective. 

This  process  shall  enable  the  evaluation  of  more 
than  one  subtask  at  a  time;  or  the  evaluation  of  an 
entire  task  when  the  task  has  been  broken  down  into 
subtasks  and  it  has  been  determined  that  an 
evaluation  shall  not  occur  for  each  subtask. 

Before  a  test  item  can  be  selected  for  a  given 
objective,  the  test  item  must  first  reside  in  the 
Test  Item  Bank.  Capabilities  shall  be  provided 
to  support  the  selection  of  appropriate  types  of 
test  items  for  a  given  test. 

At  least  one  OTG  and  one  PEC  must  be  selected 
for  the  development  of  a  performance  test. 

-  At  least  one  knowledge  test  item  must  be 
selected  for  the  development  of  a  knowledge 
test. 
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DESCRIPTION: 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Select  Objectives  and  Test  Items  (continued) 


(continued) 

For  a  each  selected  objective  for  a  performance 
test: 


1.  All  test  items  (OTGs  and  PECs)  maintained  in 
the  Test  Item  Bank  for  the  objective  shall  be 
automatically  called  up;  the  system  shall 
display  the  Test  Item  Id  for  each  test  item. 

2.  The  test  developer  shall  be  able  to  review 
each  test  item  and/or  select  those  test 
items  to  be  incorporated  into  the  test. 

3.  Highlighting,  or  some  other  identifiable 
means,  shall  be  used  to  designate  each  test 
item  selected  to  be  incorporated  into  the 
test. 

For  each  selected  objective  for  a  knowledge  test, 
the  test  developer  shall  be  able  to: 

1.  Review/ select  specific  knowledge  test  items, 

2.  Review/select  all  knowledge  test  items, 

3.  Select  a  random  number  of  knowledge  test 
items ,  or 

4.  Select  random  presentation  of  test  items. 
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PROCESS : 

DESCRIPTION: 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Select  Objectives  and  Test  Items  (continued) 


(continued) 

When  the  test  developer  elects  specific  test 
items  or  all  test  items:  all  knowledge  test  items 
maintained  in  the  Test  Item  Bank  for  the  selected 
objective  shall  be  automatically  called  up;  the 
system  shall  display  the  Test  Item  ID  that 
applies  to  each  test  item.  The  test  developer 
shall  be  able  to  select  one  or  more  of  the  test 
items  which  have  been  called  up,  and  may  review 
each  test  item  whether  or  not  the  item  is 
selected  for  the  test.  Highlighting,  or  some 
other  identifiable  means,  shall  be  used  to 
designate  each  item  that  is  selected  to  be 
incorporated  into  the  test. 

When  the  test  developer  elects  a  random 
number  of  knowledge  test  items:  the  system  shall 
automatically  select  the  random  test  items  based 
on  the  number  of  test  items  the  test  developer 
defines  for  the  test  and  the  total  knowledge  test 
items  maintained  in  the  Test  Item  Bank  for  the 
selected  objective.  The  system  shall  display  the 
Test  Item  ID  of  each  selected  test  item.  The 
test  developer  shall  be  able  to  review/accept  the 
randomly  selected  items,  or  de-select  one  or  more 
items  and  select  other  items  maintained  for  that 
ob j  ective . 

When  the  test  developer  elects  to  randomly  select 
test  items  at  time  of  presentation:  the  system 
shall  randomly  select  the  test  items  at  the  point 
in  time  when  the  test  is  either  administered 
on  line  (Process  2.2.1)  or  printed  for  off-line 
administration  (Process  2.2.2).  The  number  of 
items  randomly  selected  shall  be  based  on  the 
number  of  test  items  the  test  developer  defines 
for  the  test  and  the  total  test  test  items 
maintained  in  the  Test  Item  Bank  for  the  selected 
objective. 
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PROCESS : 

DESCRIPTION: 


DATA  INPUT: 

DATA  OUTPUT: 

STRUCTURE: 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Select  Objectives  and  Test  Items  (continued) 


(continued) 

Once  a  test  is  developed,  a  test  reviser  shall 
be  able  to: 

1.  Review  the  list  of  selected  objectives, 

2.  Review  the  list  of  selected  test  items  for 
each  objective, 

3.  Review  each  selected  test  item, 

4.  De-select  any  test  item  previously  selected, 

5.  De-select  any  objective  previously  selected 
(which  will  cause  the  respective  selected 
test  items  to  be  automatically  deleted) ,  or 

6.  Select  additional  objectives  and/or  test 
items . 

NOTE:  These  capabilities  for  revising  a  test 

shall  not  apply  for  a  knowledge  test  when 
the  test  items  are  randomly  selected  at 
time  of  presentation. 


AFS  Task  Data 
BO  Data 

Test  Item  Data 

Selected  Objectives  &  Test  Items 
Task  Evaluation  Strategy  Info 


Selected  Objectives  &  Test  Items 


N/A 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


PROCESS:  2. 1.3. 2. 1.2,  Input  Test  Format 


LEVEL  OF  AUTOMATION:  Automated,  automatic 


DESCRIPTION:  This  process  shall  enable  a  test  developer/reviser 

to  format  a  test.  The  format  of  the  test  reflects 
the  order  in  which  the  test  items  will  be  presented 
when  the  test  is  administered. 

A  test  shall  be  automatically  formatted  as  test 
items  are  selected  when  developing/revising  the 
test.  The  test  items  shall  be  ordered  in  the 
sequence  selected.  The  test  developer  or  reviser 
shall  be  able  to  accept  the  automatic  format  or 
re-order  the  test  items  to  achieve  the  desired 
format. 


In  addition  to  reordering  test  items,  this  process 
shall  also  enable  a  test  developer/reviser  to 
designate  critical  knowledge  test  items.  Critical 
knowledge  test  items  are  items  that  are  considered 
essential  to  task  competency;  if  a  trainee 
incorrectly  responds  to  a  critical  knowledge  test 
item,  the  entire  knowledge  test  is  failed. 

Critical  knowledge  test  items  shall  be  highlighted 
or  identified  in  some  other  way. 


DATA  INPUT :  Test  Format  Data 


DATA  OUTPUT:  Test  Format  Data 


STRUCTURE :  N/A 
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PROCESS:  2. 1.3. 2.1.3,  Input  Test  Parameters 

LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  developer  or 

reviser  to  manipulate  the  following  three  test 
parameters  for  a  performance  test: 

1.  test  use 

2 .  item  analysis 

3.  maximum  analysis  samples. 

The  process  shall  enable  a  test  developer/reviser 
to  manipulate  the  following  seven  test  parameters 
for  a  knowledge  test: 


1. 

time  allowed 

2. 

test  use 

3. 

item  analysis 

4. 

maximum  analysis 

samples 

5. 

item  scrambling 

6. 

test  interruption 

(on 

line 

only) 

7. 

test  item  recap 

(on 

line 

only) 

The  parameters  shall  be  defined,  as  follows: 

Time  Allowed:  The  test  developer/reviser  may  set 
a  maximum  time  limit  in  minutes  for  the  test  taker 
to  complete  the  test  on  line.  The  default  setting 
shall  be  60  minutes?  however,  the  developer  may 
set  any  limit  between  1  and  1440  minutes. 

Test  Use:  The  test  developer/reviser  may 
designate  a  performance  or  knowledge  test  as  being 
the  primary  test  of  its  type  or  an  alternate  test. 
Primary  shall  be  the  default  setting  for  the  first 
test  of  its  type  (performance  or  knowledge)  that 
is  developed  for  a  behavioral  objective. 

Alternate  shall  be  the  default  setting  for  each 
test  subsequently  developed  for  the  same 
objective. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Input  Test  Parameters  (continued) 


(continued) 

Item  Analysis:  The  test  developer/ reviser  may 
enable  or  disable  the  item  analysis  option. 

Enabled  shall  be  the  default  setting.  When  item 
analysis  is  enabled,  the  test  item  analysis 
process  (2. 1.3. 2. 2)  shall  run  automatically 
whenever  the  specified  number  of  samples  of  data 
have  been  accumulated.  (NOTE:  The  item  analysis 
function  must  be  disabled  before  the  test 
developer  shall  be  able  to  invoke  the  Select 
Random  Items  at  Presentation  option  in  the 
selection  of  knowledge  test  items.) 

Maximum  Analysis  Samples:  The  test  developer 

or  reviser  may  specify  the  maximum  number  of  test 
samples  to  be  stored  for  analysis.  The  test 
developer/ reviser  may  select  any  number  from  30  to 
100.  The  default  setting  shall  be  50. 

Item  Scrambling:  The  test  developer/reviser  may 
designate  whether  the  test  items  are  to  be 
presented  to  the  test-taker  in  the  order  displayed 
in  the  test  format  or  are  to  be  automatically 
scrambled  for  presentation.  The  default  setting 
shall  be  disabled. 

Test  Interruption:  The  test  developer/reviser  may 
determine  whether  a  test-taker,  continuing  to  take 
an  on-line  test  after  an  interruption  of  15 
minutes  or  more,  shall  be  permitted  to  review  test 
items  and  change  responses  to  items  presented 
prior  to  the  interruption.  The  default  setting 
shall  be  allowed.  When  an  interruption  is  less 
than  15  minutes,  review  and  changing  of  responses 
shall  always  be  allowed. 

Test  Item  Recap:  The  test  developer/reviser  may 
determine  whether  the  test-taker,  taking  a  test 
online,  shall  be  allowed  a  recapitulation  of  the 
items  that  were  incorrectly  answered  on  the  test. 
The  default  setting  shall  be  allowed. 
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PROCESS : 

DATA  INPUT: 

DATA  OUTPUT: 

STRUCTURE : 


Input  Test  Parameters  (continued) 
Test  Parameter  Data 

Test  Parameter  Data 

N/A 
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PROCESS:  2. 1.3. 2. 1.4,  Input  Knowledge  Test  Instructions 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  test  developer  or 

reviser  to  input  and  revise  knowledge  test 
instructions . 

Test  instruction  templates  shall  be  developed  and 
automated  and  shall  consist  of  general  instructions 
for  taking  a  knowledge  test.  A  test  developer  or 
reviser  shall  be  able  to  add,  revise  or  delete  data 
to  accommodate  instructions  for  each  specific 
knowledge  test. 

There  shall  be  two  knowledge  test  instruction 
templates;  one  for  on-line  testing,  the  other  for 
off-line  testing.  The  test  developer  shall 
indicate  whether  the  test  is  to  be  administered 
only  on  line,  or  both  on  line  and  off  line.  The 
template (s)  corresponding  to  the  test  developer's 
decision  shall  be  automatically  called  up  for 
editing. 


DATA  INPUT:  Knowledge  Test  Instructions 


DATA  OUTPUT:  Knowledge  Test  Instructions 


STRUCTURE :  N/A 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


PROCESS :  2 . 1 . 3 . 2 . 2 ,  Analyze  Evaluation  Materials 


LEVEL  OF  AUTOMATION:  Automatic,  automated 


DESCRIPTION:  This  automatic  process  shall  enable: 

1.  Automatic  consolidation  of  specific  test, 
test  item  and  test  score  results  data. 

2.  Automatic  sorting  of  consolidated  data  by 
test,  objective,  subtask  and  task. 

3.  Automatic,  or  as  requested,  analysis  of 
consolidated/sorted  data. 

4.  Automatic  reporting  of  the  evaluation 
analsyis  results. 


DATA  INPUT:  Test  Item  Analysis  Data 

Test  Analysis  Data 
Test  Score  Data 


DATA  OUTPUT:  Test  Item  Evaluation  (TIE)  Report 


STRUCTURE:  This  process  includes  the  following  subprocesses 

-  Process  2. 1.3. 3.1,  Consolidate  Test/Test  Item 

Analysis  Data 

-  Process  2 . 1 . 3 . 3 . 2 ,  Perform  Test/Test  Item 

Analysis 
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PROCESS:  2. 1.3. 2. 2.1,  Consolidate  Test/Test  Item  Analysis 

Data 


LEVEL  OF  AUTOMATION:  Automatic 


DESCRIPTION:  This  automatic  process  shall  consolidate  all  data 
required  for  performing  an  analysis  of  tests  and 
test  items.  Data  to  be  analyzed  result  from  a 
variety  of  processes: 

1.  As  each  test  item  is  developed  or  revised, 
the  appropriate  test  item  analysis  data  (test 
item  ID,  correct  answer,  etc)  shall  be 
automatically  stored  for  future  analysis. 

2.  As  each  test  is  developed  or  revised,  the 
appropriate  test  analysis  data  (Test  ID, 
test  format,  etc.)  shall  be  automatically 
stored  for  future  analysis  purposes. 

« 

3.  As  each  test  is  administered  and  scored, 
the  trainee's  responses  and  overall  test 
results  (pass/ fail)  shall  be  automatically 
stored  for  future  analysis. 

Test/Test  Item  analysis  data  shall  be  consolidated 
to  support  either  of  the  following: 

1.  Automatic  analysis  of  a  test  and  its  items, 
once  thirty  samples  of  the  test  have  been 
scored,  or 

2.  Analysis  of  a  test  and  its  items  based  on  a 
test  developer's  request  to  perform  the 
analysis.  (Data  for  all  samples,  up  to  the 
maximum  number  specified  in  the  test  format, 
shall  be  consolidated.) 
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PROCESS : 

DESCRIPTION: 


DATA  INPUT: 

DATA  OUTPUT: 

STRUCTURE: 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Consolidate  Test/Test  Item  Analysis  Data 
(continued) 


(continued) 

At  periodic  intervals  (each  time  the  final  task 
lists  are  stored  -  Process  1.1. 1.3. 6  of  the 
Management  Subsystem)  the  system  shall 
automatically  consolidate  and  sort  identification 
data  pertaining  to  each  test.  This  process  shall 
produce  a  key  which  aligns  test  identification  data 
for  all  objectives;  the  key  shall  enable  a  test 
analyzer  to  input  specific  data  to  initiate  an 
analysis  for  any  test  listed. 


Test  Analysis  Data 
Test  Item  Analysis  Data 
Test  Score  Data 
Test  Analysis  Request  Data 


Consolidated  Analysis  Data 
Test  Item  Analysis  Key 


N/A 
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PROCESS:  2. 1.3. 2. 2. 2,  Perform  Test/Test  Item  Analysis 

LEVEL  OF  AUTOMATION:  Automatic,  automated 


DESCRIPTION:  This  process  shall  enable  independent  analysis  of 

any  test  residing  in  the  system.  This  process 
shall:  extract  the  appropriate  consolidated 
analysis  data  needed  to  analyze  a  specific  test; 
calculate  statistical  data;  and  generate  a 
document  which  reports  the  analyzed  data. 

This  process  shall  be  accomplished  in  one  of  two 
ways: 


1.  When  the  minimum  number  of  test  samples  have 
been  consolidated  for  a  particular  test,  or 

2.  When  a  test  developer  submits  an  on-line 
request  for  an  analysis  of  a  particular  test 

The  minimum  number  of  test  samples  required  for 
each  test  to  be  automatically  analyzed  shall  be 
established  as  thirty.  Once  thirty  samples  of  a 
given  test  have  been  consolidated,  the  system 
shall  calculate  the  consolidated  data  and  produce 
report  which  summarizes  the  analysis  results. 

A  test  analyzer  shall  be  able  to  request  an 
analysis  of  a  particular  test,  at  any  time,  using 
keyed  test  identification  data  and  specifying 
one  or  more  of  the  following  selection  criteria: 

1.  All  samples  of  the  test  during  a  specified 
timeframe, 

2.  All  samples 
failed, 

3 .  All  samples 
passed, 

4.  All  samples  of  the  test. 


of  the  test  which  have  been 
of  the  test  which  have  been 
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Perform  Test/Test  Item  Analysis  (continued) 


(continued) 

Once  the  test  analyzer  has  input  the  test 
identification  data  (from  the  key),  and  entered 
one  or  more  of  the  item  analysis  selection 
criteria,  the  system  shall  automatically  calculate 
the  consolidated  data  and  produce  a  report  which 
summarizes  the  analysis  results. 

This  process  shall  generate  a  document  that  reports 
test,  subscale  (objective)  and  test  item  analysis 
results.  To  obtain  further  information  regarding 
the  types  of  analysis  results  which  are  reported, 
refer  to  the  description  of  the  data  flow  titled: 
Test  Item  Evaluation  (TIE)  Report. 


Test  Item  Analysis  Keys 
Consolidated  Analysis  Data 


Test  Analysis  Reguest  Data 
Test  Item  Evaluation  (TIE)  Report 

N/A 


105 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  1  of  2 


PROCESS:  2. 1.3. 2. 3,  Validate  Evaluation  Materials 

LEVEL  OF  AUTOMATION:  Manual 


DESCRIPTION:  This  off-line  process  shall  enable  a  test 

developer  to  validate  evaluation  materials  against 
a  criterion  of  job  performance  in  an  operational 
setting  IAW  Air  Force  Instructional  Systems 
Development  (ISD)  policy. 

This  process  shall  provide  for  identification  of: 
changes  in  operational  requirements,  changes  in 
task  performance  and  proficiency  documentation 
(publications) ,  and  test/ test  item  analysis 
results,  which  warrant  review  and  possible 
revision  of  existing  task  (or  subtask)  evaluation 
materials. 

The  test  developer  shall  apply  subject  matter 
expertise  and  knowledge  of  ISD  principles  and 
procedures  in  order  to  make  an  informed  judgment 
concerning  the  adequacy  of  a  particular  test 
and  test  items.  The  developer  shall  determine 
whether  the  test(s)  for  a  particular  behavioral 
objective  is/are  adequate  to  measure  the 
competency  of  the  trainee's  knowledge,  skill  and 
performance  capabilities  for  a  given  subtask  or 
task. 

When  this  process  is  performed,  each  test  and  its 
test  items  shall  be  scrutinized  with  respect  to 
the  following  concerns: 

1.  Technical  accuracy  (does  the  test  contain 
only  accurate  technical  data) ; 

2.  Technical  adequacy/validity  (are  the  right 
questions  being  asked;  are  the  right 
performance  steps  being  observed) ; 

3.  Discrimination  (does  the  item  differentiate 
between  airmen  who  have  mastered  the 
objective  and  airmen  who  have  not) ; 
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Validate  Evaluation  Materials  (continued) 


(continued) 

4.  Format  (are  the  test  and  item  format 
consistent  with  accepted  evaluation 
procedures) ; 

5.  Instructions  (are  the  instructions  clear 
and  concise) ; 

6.  Time  (is  sufficient  time  allowed  to  complete 
the  test) ; 

7.  Reliability  (all  things  being  equal,  will 
the  same  results  be  obtained  each  time  the 
test/ item  is  presented  to  the  same 
individual) . 

If  any  of  the  representative  skills  or  knowledge 
for  task  performance  are  not  adequately  tested  by 
one  or  more  test  items,  the  development  of 
additional  tests  and/or  test  items  or  the  revision 
of  tests/test  items  shall  be  indicated. 


AFS  Task  Data 
Publications 
BO  Data 
Test  Data 
Test  Item  Data 

Task  Evaluation  Strategy  Info 
Test  Item  Evaluation  (TIE)  Report 
Operational  Feedback 
Publication  Change  Notice 


Test  Revision  Requirement (s) 

Test  Item  Revision  Requirement (s) 


N/A 
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DATA  FLOW:  APS  Task  Data 


DESCRIPTION:  An  aggregation  of  task  record  data  for  a  specific 

task.  The  data  result  from  a  task  analysis  and 
are  updated  on  line,  via  processes  within  the 
Management  Subsystem.  These  data  shall  be 
available  for  on-line  review  or  may  be  printed. 
AFS  task  data  shall  be  reviewed  and  used  to: 

-  Develop/ revise  one  or  more  behavioral 
objectives  for  the  task  and  its  subtasks 
(if  subtasks  exist) , 

-  Plan  evaluation  materials  for  the  task, 

-  Develop/revise  test  items  and  tests  which 
shall  be  used  to  evaluate  trainees  knowledge 
and  performance  capabilities  regarding  the 
task,  and 

-  Validate  each  test  (and  the  respective  test 
items)  administered  and  scored  for  the  task. 


DATA  SOURCE: 


External  Entity  l.l 


DATA  DESTINATION: 


DFD  2.1,  Process  1,  2,  3 

DFD  2.1.1,  Process  1,  2 

DFD  2.1.2,  Process  l 

DFD  2.1.3,  Process  l,  2 

DFD  2 . 1 . 3 . 1 ,  Process  1 

DFD  2. 1.3. 1.1,  Process  1,  2,  3 

DFD  2. 1.3. 2,  Process  1,  3 

DFD  2. 1.3. 2.1,  Process  1 
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AFS  Task  Data  (continued) 


Data  elements  shall  include: 


Description 

The  Air  Force  Specialty  Code  for  which 
the  task  applies.  AFSCs  are  identified 
by  an  alphanumeric  string;  the  basic 
AFSC  is  identified  by  a  string  of  five 
numerals;  however,  an  AFSC  may  have  an 
alpha  prefix  and/or  a  numerical  suffix. 
Data  Type:  alpha/ numeric  string  5-7 
characters. 

The  code  that  uniquely  identifies  the 
task  from  other  tasks  in  the  AFSC.  The 
first  digit  is  always  alphabetical  (a-z) 
and  represents  the  task  category  for 
which  the  task  applies.  The  last  five 
digits  are  always  numerical  (00000- 
99999) . 

Data  Type:  Alpha/numeric  string  of  6 
characters . 

A  numeric  identification  of  the  version 
of  the  task.  (The  version  number  shall 
increase  by  one  each  time  an  AOTS 
software  installation  occurs  if  certain 
data  for  the  task  has  been  modified 
since  the  last  installation.) 

Data  Type:  Integer  (up  to  2  digits) . 

The  numeric  identifications  of  each 
subtask  applicable  to  the  task. 

(Subtasks  are  numbered  automatically  and 
consecutively;  up  to  25  subtasks  may 
exist  for  the  task.) 

Data  Type:  Integer  (range  1-25) . 
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DATA  FLOW:  AFS  Task  Data  (continued) 


STRUCTURE: 


(continued) 


Data  Items 

AOTS  Task  Statement 


Weapon  Systems/ 
Equipment  Supported 


Other  Systems  Data 
*  -  Other  System  Names 

-  Other  System  Task 


Descriptions 

Statement  which  reflects  what  must  be 
accomplished.  May  be  identical  to  an 
OSR  statement,  a  revision  of  an  OSR 
statement,  or  a  statement  created  by 
a  Subject  Matter  Expert  (SME) .  The 
task  statement  shall  contain  an  active 
verb  and  an  object. 

Data  Type:  Alpha/ numeric  string,  not  to 
exceed  240  characters/ spaces. 

Name  of  the  weapon  systems/ equipment 
that  the  task  is  performed  on  (RF-4C 
aircraft,  J79-15  engine,  etc.). 

Data  Type:  Array  (1-X  system/equipment 
names) . 


Name  of  each  automated  system,  such  as 
CAMS,  for  which  data/ automated  processes 
exist  for  the  task. 

Data  Type:  Array  (1-X  names). 

Identification  characters  used  in  the 
other  automated  systems  for  the  task. 
Data  Type:  Array  (1-X  IDs). 


Corresponding  STS 


A  cross-reference  depicting  one  or  more 
Specialty  Training  Standard  (STS) 
in  which  the  task  is  listed. 

Data  Type:  Array  (1-X  STSs) 
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DATA  FLOW:  AFS  Task  Data  (continued) 


STRUCTURE:  (continued) 


Data  Items 


Descriptions 


Task  References 


Performance  Resources 


Prerequisite  Tasks 


Task  Performance 
Location (s) 


Supporting  Knowledge 
and  Skills 


Task/Subtask 

Activities 


Titles,  identification  codes,  and 
breakdowns  of  the  publications  that 
specify  performance  and  proficiency 
requirements  for  the  task. 

Data  Type:  Array  (1-X  references) ; 

Array  (1-X  breakdowns) . 

Listing  of  resources  (tools,  equipment, 
and  materials)  required  for  satisfactory 
task  performance. 

Data  Type:  Array  (1-X  resources) . 

Task  IDs  of  tasks  that  must  be 
accomplished  prior  to  or  in  conjunction 
with  the  task  under  consideration. 

Data  Type:  Array  (1-X  tasks) . 

Alpha/numerically  coded  identification 
of  the  location (s)  —  base,  unit,  and 
workcenter  —  at  which  the  task  is 
performed. 

Data  Type:  Array  (1-X  bases) ;  Array 
(1-X  units);  Array  (1-X  workcenters) . 

Listing  of  the  prerequisite  knowledge 
and  skills  for  satisfactory  task 
performance. 

Data  Type:  Array  (1-X  knowledge/skills) 

The  actions  or  steps  (usually  but  not 
necessarily  sequential)  required  for 
satisfactory  task  performance. 

Data  Type:  Array  (1-X  activities) . 
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DATA  FLOW:  Bad  Verbs 


DESCRIPTION:  Unacceptable  (inactive)  verbs  which  are 

automatically  matched  against  the  verb  in  each 
newly  developed  or  revised  behavioral  objective 
statement.  Bad  verbs  (up  to  100)  shall  reside  on 
a  list;  each  verb  shall  be  examined  and  matched  to 
each  BO  statement  under  development/revision. 

When  a  BO  statement  verb  matches  a  bad  verb,  the 
statement  shall  be  rejected  and  the  BO  developer 
or  reviser  shall  be  required  to  reenter  an 
acceptable  verb  for  the  BO  statement. 


DATA  SOURCE: 


DFD  2.1, 

DFD  2.1.1, 
DFD  2. 1.1.1, 


Process  1  and 
Process  3  and 
Data  Store  E2 


Data  Store  E2 
Data  Store  E2 


DATA  DESTINATION: 


DFD  2.1,  Data  Store  E2 
DFD  2.1.1,  jacd  Store  E2 
DFD  2. 1. i. x, Process  2 


and  Process  1 
and  Process  1, 


2, 


3 


STRUCTURE:  N/A.  Data  Type:  Array  (1-X  bad  verbs) ,  where 

X  cannot  exceed  iGO.  Each  bad  verb  within  the 
array  is  an  alpha  string,  not  to  exceed  20 
characters/ spaces . 
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DATA  FLOW:  BO  (Behavioral  Objective)  Data 


DESCRIPTION:  Data  which  have  been  developed  or  revised  for  a 

given  behavioral  objective.  A  comprehensive 
behavioral  objective  shall  consist  of  a  complete 
behavioral  objective  statement;  the  references 
(publications  and  publications  breakdowns)  which 
set  forth  task  performance  and  proficiency 
requirements  for  the  objective;  the  resources 
(performance,  training  and  evaluator)  required  for 
satisfactory  accomplishment  of  the  objective;  and 
a  list  of  the  training  materials  which  apply  to 
the  objective. 


DATA  SOURCE:  DFD  2.1,  Process  1  and  Data  Store  El 

DFD  2.1.1,  Process  1  and  Data  Store  El 

DFD  2.1.2,  Data  Store  El 

DFD  2.1.3,  Data  Store  El 

DFD  2. 1.3.1,  Data  Store  El 

DFD  2. 1.3. 1.1,  Data  Store  El 

DFD  2. 1.3. 2,  Data  Store  El 

DFD  2 . 1 . 3 . 2 . 1 ,  Data  Store  El 


DATA  DESTINATION:  DFD  2.1,  Data  Store  El  and  Process  1,2,3 

DFD  2.1.1,  Data  Store  El  and  Process  2 

DFD  2.1.2,  Process  1 

DFD  2.1.3,  Process  1 , 2 

DFD  2. 1.3.1,  Process  1 

DFD  2 . 1 . 3 . 1 . 1 , Process  1 ,  2 ,  3 

DFD  2. 1.3. 2,  Process  1,  3 

DFD  2.1. 3.2.1, Process  1 
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DATA  FLOW:  BO  Data  (continued) 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 


Description 


Objective  ID  A  unique  number  identifying  the 

particular  behavioral  objective  for 
which  the  data  applies.  This  ID  is 
automatically  assigned  by  the  system 
once  the  BO  statement  is  developed,  and 
cannot  be  revised. 

Data  Type:  Integer  (number  is  limited 
only  by  system's  capacity  for  relative 
records  per  file) . 


Complete  Behavioral  Objective  Statement 


-  Behavioral 
Component 


A  brief  statement  headed  by  an  active 
verb  that  specifies  the  required 
behavior. 


-  Conditions 
Component 


The  operational  conditions  under  which 
the  required  behavior  will  normally 
occur. 


-  Standards  A  measurable  minimum  level  of 

Component  performance  that  shall  be  considered  to 

demonstrate  an  acceptable  level  of 
competency . 

Data  Type  for  complete  statement: 
alpha/numeric  string,  not  to  exceed  240 
characters/spaces . 
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DATA  FLOW: 


BO  Data  (continued) 
(continued) 


BO  References  Data 


-  Publications  Publications  listed  by  identification 

code  and  and  title  that  define  the 
performance  and  proficiency 
requirements  for  the  objective. 

Data  Type:  Array  (1-X  publication 
IDs/titles?  where  X  cannot  exceed  6) . 

-  Breakdowns  Breakdowns  (by  section,  chapter,  figure, 

table,  page,  or  other  part)  of  the 
publications  that  define  the  performance 
and  proficiency  requirements  for  the 
objective. 

Data  Type:  Array  (1-X  breakdowns;  where 
X  cannot  exceed  100) . 

BO  Resources  List  of  tools,  equipment  and  material 

required  for  successful  performance  of 
the  objective  (called  performance 
resources) ,  as  well  as  the  training  and 
evaluator  resources  required  to 
accomplish  the  objective. 

Data  Type:  Array  (1-X  performance 
resources) ;  Array  (1-X  evaluator 
resources) ;  Array  (1-X  training 
resources);  where  X  cannot  exceed  30. 

BO  Training  Materials  List  of  training  materials  which  exist 

for  accomplishing  the  objective.  These 
materials  can  be  of  many  types:  Computer 
Assisted  Instruction  (CAI)  lessons, 
films,  sound-on-slide  presentions,  etc. 
Data  for  each  material  includes:  Name 
of  item,  type  of  item  and  other 
identification  data  (course  ID,  film 
number/ ID ,  etc . ) 

Data  Type:  Array  (1-X  materials),  where 
X  cannot  exceed  15. 
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DATA  FLOW:  BO  Data  for  Event  Scheduling 


DESCRIPTION:  Data  applying  to  a  particular  behavioral 

objective,  which  is  passed  to  the  Management 
Subsystem  to  enable  the  scheduling  of  a 
training  or  evaluation  event.  These  data  shall 
support  either  a  task  knowledge  training  or 
evaluation  event;  a  task  performance  training 
or  evaluation  event;  or  a  Quality  Contol  (QC) 
evaluation  event. 


DATA  SOURCE: 


DFD  2.1,  Data  Store  El 
DFD  2.1.1,  Data  Store  El 


DATA  DESTINATION:  External  Entity  1.3 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 
Objective  ID 


BO  Resources 


Description 

A  unique  number  identifying  the 
particular  behavioral  objective  for 
which  the  BO  data  applies.  This  ID  is 
automatically  assigned  by  the  system 
once  the  BO  statement  is  developed. 

Data  Type:  Integer  (number  is  limited 
only  by  system's  capacity  for  relative 
records  per  file) . 

The  performance  and  training  or 
evaluator  resources  which  have  been 
specified  for  the  objective. 

Data  Type:  Array  (1-X  performance 
resource  names  and  descriptions) ;  Array 
(1-X  training  or  evaluator  resource 
names  and  descriptions) ;  where  X  cannot 
exceed  30. 
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DATA  FLOW:  BO  ID 


DESCRIPTION  A  unique  identification  number  assigned  by  the 

system  to  a  particular  behavioral  objective,  once 
the  respective  BO  statement  has  been  developed.  * 
Each  BO  ID  shall  be  used  to  key  (link) : 

-  Data  for  a  given  behavioral  objective 
(i.e.,  BO  statement,  references,  resources 
and  training  materials) , 

-  The  behavioral  objective  to  the  respective 
task  or  subtask, 

The  behavioral  objective  to  each  test  and 
test  item  which  is  developed  for  evaluating 
the  accomplishment  of  the  objective. 


DATA  SOURCE :  DFD  2.1.,  Process  1 

DFD  2.1.1,  Process  1 
DFD  2. l.l.l.  Process  2 


DATA  DESTINATION  DFD  2.1,  External  Entity  1.1  and  1.3 

DFD  2.1.1,  External  Entity  1.1  and  1.3 

DFD  2. 1.1.1,  External  Entity  1.1  and  1.3  and 

Data  Store  El 

STRUCTURE:  H/A.  Data  Type:  Integer  (number  is  limited  only 

by  system's  capacity  for  relative  records  per 
file) . 
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DATA  FLOW:  BO  References 


DESCRIPTION:  Official  publications  and  the  breakdowns  (by 

section,  chapter,  paragraph,  table,  page  number, 
etc.)  that  define  the  performance  and  proficiency 
requirements  for  a  given  behavioral  objective. 

BO  references  may  only  be  selected  from  the 
respective  task/ subtask  references,  and  BO 
reference  breakdowns  shall  apply  only  to  the 
selected  BO  references.  BO  references  shall  be 
used  to  select  respective  test  item  references. 


DATA  SOURCE: 


DFD  2. 1.1.1,  Process  3 
DFD  2. 1.3.1,  Data  Store  El 


DATA  DESTINATION: 


DFD  2. 1.1.1,  Data  Store  El 
DFD  2. 1.3.1,  Process  2 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 

Selected  References 


Reference  Breakdowns 


Description 

Publications,  listed  by  identification 
code  and  title,  that  have  been  selected 
from  the  task/ subtask  references; 
thereby  designating  them  as  the  BO 
references . 

Data  Type:  Array  (1-X  reference  names 
and  titles),  where  X  cannot  exceed  6. 

Breakdowns  (by  section,  chapter,  figure, 
table,  page,  or  other  part)  of  a 
publication  that  define  the  performance 
and  proficiency  requirements  for  the 
objective. 

Data  Type:  Array  (1-X  breakdowns) , 
where  X  cannot  exceed  100. 
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DATA  FLOW:  BO  Resources 


DESCRIPTION:  The  resources  which  apply  to  a  given  behavioral 

objective.  These  resources  shall  include: 

Performance  resources  (tools,  equipment  and 
materials)  which  are  required  for 
accomplishing  the  object.  The  BO  performance 
resources  shall  be  selected  from  those 
resources  identified  for  the  task  or  subtask 
to  which  the  BO  applies;  thereby  designating 
them  as  the  BO  performance  resources. 

Training  resources  (slide  projector, 
chalkboard,  etc.)  which  are  required  when 
training  someone  to  accomplish  the  objective. 

-  Evaluator  resources  (stopwatch,  clipboard) 
which  are  required  to  evaluate  someone 
accomplishing  the  objective. 


BO  Performance  A  list  of  performance  resources 

Resources  identified  for  the  behavioral  objective. 

Data  Type:  Array  (1-X  resource  names 
and  descriptions) ,  where  X  cannot  exceed 
30. 


BO  Training  Resources  A  list  of  training  resources  identified 

for  the  behavioral  objective. 

Data  Type:  Array  (1-X  resource  names 
and  descriptions) ,  where  X  cannot  exceed 
30. 
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DATA  FLOW: 

STRUCTURE : 
Data  Items 
BO  Evaluator 
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BO  Resources  (continued) 


(continued) 


Descriptions 

Resources  A  list  of  evaluator  resources  identified 
for  the  behavioral  objective. 

Data  Type:  Array  (1-X  resource  names 
and  descriptions) ,  where  X  cannot  exceed 
30. 
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Date:  31  October  1989 


DATA  FLOW:  BO  Training  Materials 


DESCRIPTION:  Training  materials  (Computer  Assisted  Instruction 

lessons,  publications,  filmstrips,  audio  tapes, 
etc.)  that  apply  to  a  given  behavioral  objective. 
The  maximum  number  of  training  materials  per 
objective  is  15. 


DATA  SOURCE: 


DFD  2. 1.1.1,  Process  5 


DATA  DESTINATION:  DFD  2. 1.1.1,  Data  Store  El 


STRUCTURE:  Data  elements  shall  include  (for  each  material 

listed  for  the  behavioral  objective) : 


Data  Item 

Type  of  Training 
Material 


Training  Material  ID 


Training  Material 
Description 


Description 

Data  which  identifies  the  training 
material  type.  The  type  shall  be  one 
of  the  following:  C*CAI  lesson, 

T~Taxt ,  F*film,  S»Sound-on-Slide,  and 
O-Other. 

Data  Type:  Alpha  string  of  l  character. 

Data  identifying  the  training  material. 
This  shall  be  a  CAI  lesson  number, 
a  training  handbook  number,  etc. 

Data  Type:  Alpha/ numeric  string,  not 
to  exceed  16  characters/ spaces. 

Data  describing  the  training  material, 
such  as  the  name  of  a  film,  the  title 
of  a  CAI  lesson,  etc. 

Data  Type:  Alpha/numeric  string,  not 
to  exceed  50  characters/ spaces. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Complete  BO  Statement 


DESCRIPTION:  A  complete  behavioral  objective  statement 

consisting  of  the  behavior,  the  conditions  under 
which  the  objective  will  be  performed,  and  the 
minimum  standards  for  satisfactory  performance. 
The  statement  components  (behavior,  conditions 
and  standards)  may  be  developed  separately  and 
automatically  combined  to  form  the  complete  BO 
statement,  or  the  complete  BO  statement  may  be 
developed  in  one  step. 


DATA  SOURCE:  DFD  2. 1.1.1,  Process  2 


DATA  DESTINATION:  DFD  2. 1.1.1,  Data  Store  El 


STRUCTURE:  Data  elements  shall  include: 

P3ta  ItQB  Description 

Behavioral  Component  A  brief  statement  containing  an  active 

verb  and  an  object  that  specifies  the 
required  behavior. 

Data  Type:  Alpha/numeric  string,  not 
to  exceed  80  characters/spaces. 

Conditions  Component  The  operational  conditions  under  which 

the  required  behavior  will  normally 
occur. 

Data  Type:  Alpha/ numeric  string,  not 
to  exceed  80  characters/ spaces. 

Standards  Component  A  measurable  minimum  level  of 

performance  that  shall  be  considered  to 
demonstrate  an  acceptable  level  of 
performance. 

Data  Type:  Alpha/numeric  string,  not 
to  exceed  80  characters/spaces. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Consolidated  Analysis  Data 


DESCRIPTION:  A  consolidation  of  test  analysis  data,  test  item 

analysis  data  and  test  score  data  applicable  to 
a  given  test.  Data  for  each  sample  of  the  test 
is  consolidated,  up  to  the  designated  maximum  of 
samples  for  the  test.  These  data  shall  be  used, 
along  with  automated  algorithms,  to  perform  an 
analysis  of  the  test  and  its  items. 


DATA  SOURCE:  DFD  2. 1.3. 2. 2,  Process  1 


DaTA  DESTINATION:  DFD  2. 1.3. 2. 2,  Process  2 


STRUCTURE : 


Data  elements  shall  include: 


See  data  items/descriptions  for  Data  Flows:  Test  Analysis  Data, 
Test  Item  Analysis  Data  and  Test  Score  Data.  This  data  flow  is 
an  aggregation  of  these  three  data  flows. 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Graphic (s) 


DESCRIPTION:  One  or  more  developed  on-line  graphics  which  are 

used  as  part  of  the  stem  for  a  specific  knowledge 
test  item.  Up  to  three  graphics  can  be  used  for 
any  knowledge  test  item.  Once  displayed  to  the 
test  item  developer/ reviser ,  a  graphic  may  be 
sized,  colored,  rotated  or  moved  on  the  screen  to 
enable  desired  presentation  of  the  test  item  stem. 


DATA  SOURCE:  External  Entity  5.1 


DATA  DESTINATION:  DFD  2.1,  Process  3 

DFD  2.1.3,  Process  1 
DFD  2. 1.3.1,  Process  1 
DFD  2. 1.3. 1.1,  Process  3 


Graphic 


Each  actual  graphic  identified  by  the 
developer  for  the  knowledge  test  item. 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Graphic (s)  ID(s) 


DESCSRIPTION:  The  alphanumeric  name(s)  or  code(s)  identifying 

one  or  more  graphics  to  be  included  in  a  given 
knowledge  test  item.  Each  ID  is  specified  by  a 
test  item  developer/reviser  and  is  passed  to  the 
Training  Development  Component  to  display  the 
respective  graphic  when  accomplishing  test  item 
deve 1 opment/ rev i s i on . 


DATA  SOURCE: 


DFD  2.1. 

DFD  2.1.3, 

DFD  2. 1.3.1, 
DFD  2. 1.3. 1.1, 


Process  3 
Process  1 
Process  1 
Process  3 


DATA  DESTINATION:  External  Entity  5.1 


STRUCTURE:  Data  elements  include: 

Data  Item  Description 

Each  graphic  name/code  identified  by  the 
developer  or  reviser  for  the  knowledge  test 
item  (maximum  IDs  per  test  item  is  3) . 

Data  Type:  Alpha/ numeric  string,  not  to 
exceed  10  characters/ spaces. 


Graphic  ID 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 

Page  1  of  2 


DATA  FLOW:  Knowledge  Test  Instructions 


DESCRIPTION:  Instructions  to  an  examinee  for  taking  a  specific 

knowledge  test  on  line,  off  line  or  both  on  line 
and  off  line.  Once  developed,  these  instructions 
shall  be  stored  with  other  test  data  being  main¬ 
tained  on  line;  any  or  all  of  the  instructions  may 
be  revised  as  deemed  necessary. 


DATA  SOURCE :  DFD  2 . 1 . 3 . 2 . 1 ,  Process  4  and  Data  Store  E4 


DATA  DESTINATION:  DFD  2. 1.3. 2.1,  Data  Store  E4  and  Process  4 


STRUCTURE : 


If  the  test  is  developed  for  on-line  use,  only 
on-line  instructions  will  be  developed/revised; 
for  off  line  use,  only  off  line  instructions;  for 
both  on  line  and  off  line  use,  both  sets  of 
instructions . 


Data  Item 


Data  Type  for  each  set  of  instructions: 

Alpha/ numeric  string,  not  to  exceed  3600 
characters/ spaces.  Data  elements  shall  include: 


Test  Purpose  Text  that  states  the  purpose  for  the 

knowledge  test,  which  shall  include  the 
task/ subtask  to  which  the  test  is 
relevant.  (applies  whether  on  line  or 
off  line) 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  2  of  2 


DATA  FLOW:  Knowledge  Test  Instructions  (continued) 


STRUCTURE :  ( continued ) 

Data  Items  Descriptions 

Testing  Procedures  Text  that  states  how  to  take  the  test, 

either  on  line  or  off  line. 

-  On-line  procedures  shall  include  how 
to  select  the  correct  answer (s) ,  mark 
questions  for  review,  skip  questions 
and  review  questions  once  presented. 

-  Off-line  procedures  shall  include  . 
what  type  of  pencil  to  use,  how  to 
fill  out  the  respective  answer  sheet, 
how  to  record  responses  and  what  to 
do  with  the  evaluation  materials 
(test  copy  and  answer  sheet)  when  the 
examinee  finishes  the  test. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 

Page  1  of  2 


DATA  FLOW:  Knowledge  Test  Item 


DESCRIPTION:  A  specific  knowledge  test  item  under  development 

or  revision.  The  knowledge  test  item  shall  consist 
of  the  item  stem  (including  graphics) ,  response 
alternatives,  designated  correct  answer (s) ,  and 
feedback  comments.  (For  some  types  of  items, 
feedback  may  only  be  provided  for  an  incorrect 
response) .  Once  developed,  the  knowledge  test 
item  shall  be  stored  with  other  test  items  which 
are  being  maintained  on  line;  the  test  item  may  be 
revised  as  deemed  necessary. 


DATA  SOURCE:  -  DFD  2. 1.3. 1.1,  Process  3  and  Data  Store  E3 


DATA  DESTINATION:  DFD  2. 1.3. 1.1.,  Data  Store  E3  and  Process  3 


Item  Stem 


Data  elements  shall  include: 


Type  of  knowledge  test  item  (true/ false, 
multiple  choice,  list  multiple  choice, 
matching,  fill-in-the-blank,  touch,  or 
sequence  touch) . 

Data  Type:  Alpha  string. 

The  statement/question  setting  forth  the 
situation  to  which  an  examinee  must 
respond.  The  stem  may  contain  up  to 
three  graphics. 

Data  Type:  Alpha/ numeric  string,  not  to 
exceed  3600  characters/spaces. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  2  of  2 

DATA  FLOW:  Knowledge  Test  Item  (continued) 

STRUCTURE :  ( continued) 


Data  IteSks  Descriptions 

Graphics  Data  (optional) 


Graphic  ID(s) 


-  Graphic (s) 


-  Presentation 
Specifications 


Response  Alternatives 


Correct  Answer (s) 


Feedback 


Alphanumeric  name(s)  or  code(s)  used  to 
call  up  the  graphic (s)  selected  by 
the  test  developer/reviser  for  use  in 
the  test  item  stem.  Up  to  three 
graphic  IDs  can  be  specified. 

Data  Type:  Alpha/numeric  string,  not 
to  exceed  10'  characters/spaces. 

The  actual  graphic (s)  selected  by  the 
developer/ reviser.  Up  to  three  graphics 
can  exist  for  the  test  item  stem. 

Specifications  defined  by  the  test  item 
developer/reviser  concerning  the 
placement,  scaling,  rotation  and 
coloring  of  each  selected  graphic. 

Data  Type:  X/Y  coordinates  for 
placement  data;  numeric  (l-io)  for 
scaling  factor;  numeric  (0-360  degrees) 
for  rotation  data;  and  numeric  (0-8) 
for  color  data. 

Response  alternatives  to  the  situation 
posed  in  the  item  stem  that  are  defined 
by  the  test  item  developer/ reviser. 

Data  Type  for  alternatives  depends  on 
the  type  of  knowledge  test  item. 

Response  alternative (s)  designated  by 
the  test  item  developer/ reviser  as  the 
correct  response  to  the  item  stem. 

Data  Type  for  correct  answer (s)  depends 
on  the  type  of  knowledge  test  item. 

Comments  by  the  test  item  developer  or 
reviser  for  each  response  alternative. 
Data  Type:  Alpha/ numeric  string,  not 
to  exceed  240  characters. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301. 

Date:  31  October  1989 


DATA  FLOW:  Operational  Environment  Info 


DESCRIPTION:  Off-line  information  about  the  operational 

environment  in  the  workcenter  in  which  a 
particular  task  is  performed.  The  operational 
environment  information  will  be  used  by  an 
evaluation  materials  planner?  the  information 
will  be  considered  along  with  task  analysis  and 
behavioral  objective  data  to  determine  evaluation 
parameters  for  the  task. 


DATA  SOURCE:  External  Entity  5 


DATA  DESTINATION:  DFD  2.1,  Process  2 

DFD  2,1.2,  Process  1 


STRUCTURE:  Data  elements  shall  include: 

Data  Item  Description 

This  data  flow  is  situationally  specific;  therefore,  no  attempt 
is  made  to  specify  its  contents.  Data  Type:  verbal  or  written. 


■  Specification  Number.  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Operational  Feedback 


DESCRIPTION:  Off-line  comments,  criticisms  and  observations 

from  a  workcenter  concerning  the  fit  of  specific 
AOTS  evaluation  materials  to  the  workcenter 
situation  and  the  perceived  effectiveness  of  the 
materials  in  providing  an  accurate  assessment  of 
task  performance.  Operational  feedback  (along 
with  task  analysis,  behavioral  objective,  test 
development  and  test/test  item  analysis  data) 
shall  be  used  by  an  evaluation  materials 
developer/reviser  to  validate  the  evaluation 
materials  (tests/test  items)  for  a  given  task. 


DATA  SOURCE:  External  Entity  5 


DATA  DESTINATION:  DFD  2.1,  Process  3 

DFD  2.1.3,  Process  2 
DFD  2 . 1 . 3 . 2 ,  Process  3 


STRUCTURE:  Data  elements  shall  include: 


This  data  flow  is  situationally  specific;  therefore,  no  attempt 
is  made  to  specify  its  contents.  Data  Type:  verbal  or  written. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  1  of  2 


DATA  FLOW:  OTG 


DESCRIPTION:  An  Oral  Test  Guide  (OTG)  under  development  or 

revision.  The  OTG  shall  consist  of  instructions 
for  administering  a  task  performance  evaluation  or 
a  Quality  Control  (QC)  evaluation.  Once 
developed,  the  OTG  shall  be  stored  with  other 
test  items  being  maintained  by  the  Test  Item  Bank; 
the  OTG  may  be  revised  as  deemed  necessary. 


DATA  SOURCE:  DFD  2. 1.3. 1.1,  Process  1  and  Data  Store  E3 


DATA  DESTINATION:  DFD  2. 1.3. 1.1,  Data  Store  E3  and  Process  1 


STRUCTURE:  Data  elements  shall  include: 

Data  Item  Description 

OTG  ID  The  unique,  numeric,  test  item  ID 

automatically  established  for  the  OTG 
when  it  was  initially  developed. 

Data  Type:  Integer  (number  is  limited 
only  by  system's  capacity  for  relative 
records  per  file) . 
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-  Behavioral  The  complete  behavioral  objective 

Objective  statement  for  which  the  OTG  applies. 

Statement 


-  Evaluator  Info  Information  to  be  used  by  an  evaluator 

to  help  prepare  for  a  specific  task  or 
Quality  Control  evaluation.  Such 
information  will  include:  estimated  time 
to  conduct  the  evaluation;  prerequisite 
tasks  which  should  be  trained  and 
evaluated  before  the  evaluation  is 
conducted;  whether  the  task  is  performed 
by  one  individual  or  by  a  team;  the 
tools,  equipment  and  materials  (TEMs) 
required  by  the  evaluatee;  the  TEMs 
required  by  the  evaluator;  and  any 
conditions  under  which  the  evaluation 
must  be  conducted. 


-  Evaluator  Guidelines  for  administering  the 

Instructions  performance  evaluation,  including  the 

wording  of  the  spoken  instructions  that 
the  evaluator  will  say  to  the  airman/ 
trainee. 


Data  Type  for  OTG  Text:  Alpha/numeric 
string,  not  to  exceed  8000  characters/ 
spaces. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  '  31  October  1989 


DATA  FLOW:  PEC 


DESCRIPTION:  A  Performance  Evaluation  Checklist  under 

development  or  revision.  The  checklist  shall 
specify  the  steps  required  to  be  performed  by  a 
trainee  during  a  task  performance  evaluation 
or  a  Quality  Control  (QC)  evaluation.  The  PEC 
shall  be  used  by  the  evaluator,  to  record 
observations  of  a  trainee's  attempt  to  perform 
the  steps  listed- on  the  checklist.  Once 
developed,  the  PEC  shall  be  stored  with  other 
test  items  being  maintained  by  the  Test  Item 
Bank;  the  PEC  may  be  revised  as  deemed  necessary. 


DATA  SOURCE: 


DFD  2. 1.3. 1.1,  Process  2  and  Data  Store  E3 


DATA  DESTINATION: 


DFD  2. 1.3. 1.1,  Data  Store  E3  and  Process  2 


STRUCTURE: 
Data  Item 
PEC  ID 


PEC  Steps 


Data  elements  shall  include: 

Description 

The  unique  numeric  test  item  ID 
automatically  established  for-  the  PEC 
when  initially  developed. 

Data  Type:  Integer  (number  is  limited 
only  by  system's  capacity  for  relative 
records  per  file) . 

The  text  for  each  PEC  step.  There  shall 
be  a  maximum  of  60  steps. 

Data  Type  (for  each  step) : 

Alpha/numeric  string,  not  to  exceed  160 
characters/spaces . 
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*  Specification  Number  70S647300B 
Code  Ident. .No.  76301 
Date:  31  October  1989 

DATA  FLOW:  Publications 


DESCRIPTION:  Off-line  regulations,  manuals,  technical  orders, 

operating  instructions  and  other  types  of  publica¬ 
tions  which  contain  reference  information 
applicable  to  one  or  more  tasks.  These  publica¬ 
tions  provide  task  proficiency  and  performance 
criteria  from  which  behavioral  objectives  and  test 
items  are  developed.  Publications  are  used  to 
plan  evaluation  materials  (to  help  determine 
evaluation  strategies) ,  ’ maintain  behavioral 
objectives  and  test  items  data,  and  to  validate 
implemented  evaluation  materials. 


DATA  SOURCE:  External  Entity  2 


DATA  DESTINATION:  DFD  2.1,  Process  1,  2,  3 

DFD‘2.1.1,  Process  1,  2 
DFD  2. 1.1.1,  Process  2,  3 
DFD  2.1.2,  Process  1 
DFD  2.1.3,  Process  1,  2 
DFD  2.1. 3.1,  Process  1,  2 
DFD  2. 1.3. 1.1,  Process  1,  2,  3 
DFD  2. 1.3. 2,  Process  1,  3 
DFD  2 . 1 . 3 . 2 . 1 ,  Process  1 

STRUCTURE:  Data  elements  shall  include: 

Qa£a , Items  Descriptions 

This  data  flow  is  situationally  specific;  therefore,  no  attempt 

is  made  to  specify  its  contents. 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  1  of  2 

DATA  FLOW:  Publication  Change  Notice 


DESCRIPTION:  A  printed  notice  containing  explicit  information 

identifying  one  or  more  portions  of  one  or  more 
references  (publications)  which  have  been  changed. 
Changes  in  a  publication  may  also  require  changes 
in  data  being  maintained  by  the  AOTS,  therefore, 
the  change  notice  information  shall  be  used  to  aid 
in  determining  revisions,  for  tasks,  subtasks, 
behavioral  objectives  and  test  items  potentially 
affected  by  publication  changes.  The  potentially 
affected  tasks,  subtasks,  behavioral  objectives 
and  test  items  shall  be  specified  for  each  changed 
publication. 


DATA  SOURCE: 


External  Entity  1 . l 


DATA  DESTINATION: 


DFD  2.1, 

DFD  2.1.1, 

DFD  2.1.3, 

DFD  2. 1.3.1, 
DFD  2. 1.3. 1.1, 
DFD  2. 1.3. 2, 


Process  1, 
Process  2 
Process  1, 
Process  1, 
Process  1, 
Process  3 


2  and  3 

2 

2 

2,  3 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 


Description 


IDs  of  Changed 
Publications 


The  title  and  identification  code  of 
each  publication  that  has  been  changed 
for  which  the  AOTS  maintains  data. 

Data  Type  for  ID:  alpha/ numeric  string 
not  to  exceed  20  characters/ spaces. 

Data  Type  for  title:  alpha/numeric 
string  not  to  exceed  80  characters  and/ 
or  spaces. 
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DATA  FLOW:  Publication  Change  Notice 


STRUCTURE:  (continued) 


Data  Items 


Descriptions 


IDs  of  AOTS  data 
potentially  affected 


Changed  Publication 
Breakdowns 


The  task,  subtask,  behavioral  objective 
and  test  item  IDs  potentially  affected 
by  each  publication  change. 

Data  Types:  task  ID  is  an  alpha/numeric 
string,  not  to  exceed  6  characters;  sub¬ 
task  ID  is  an  integer  (range  1-25) ; 
behavioral  objective  ID  is  an  integer; 
test  item  ID  is  an  integer. 

Designation  of  particular  segment (s)  of 
a  publication  which  more  explicitly 
identify  the  task  performance  and 
proficiency  data  (chapters,  paragraphs, 
tables,  etc.)  which  have  been  changed. 
Data  Types:  alpha/numeric  string  not  to 
exceed  40  characters/ spaces. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 

Page  1  of  2 


DATA  FLOW:  Revised  BO  Data 


DESCRIPTION:  Modifications  that  an  evaluation  materials  reviser 

has  specified  for  a  given  behavioral  objective. 

The  revision  may  include  a  change  in  the  BO 
statement,  references,  resources  and/or  training 
materials.  The  most  recent  BO  data  are  extracted 
from  the  BO  files  and  the  revised  BO  data  replaces 
the  previously  stored  BO  data. 


DATA  SOURCE:  DFD  2.1.1,  Process  2 


DATA  DESTINATION:  DFD  2.1.1,  Data  Store  El 


STRUCTURE: 


Data  elements  shall  include: 


Revised  The  new  complete  BO  statement. 

Objective  Statement  Data  Type:  Alpha/ numeric  string,  not  to 

exceed  240  characters/ spaces . 


Revised  References  The  new  list  of  references 

(publications)  and  publication 
breakdowns  (paragraphs,  chapters,  etc) 
which  apply  to  the  behavioral  objective. 
Data  Type:  Array  (1-X  reference  names 
and  descriptions,  where  X  cannot  exceed 
6) ;  Array  (1-X  breakdowns,  where  X 
cannot  exceed  100) . 

Revised  Resources  The  new  list  of  performance,  training 

and/or  evaluator  resources  applicable 
to  the  behavioral  objective. 

Data  Type:  Array  (1-X  performance 
resources) ;  Array  (1-X  evaluator 
resources) ;  Array  (1-X  training 
resources);  where  X  cannot  exceed  30. 
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DATA  FLOW:  Revised  BO  Data  (continued) 


STRUCUTE :  ( cont inued ) 


Data  Items 


Descriptions 


Revised  Training  The  new  list  of  training  materials 

Materials  applicable  to  the  behavioral  objective. 

Data  Type:  Array  (1-X  training 
materials  types,  IDs  and  descriptions) , 
where  X  cannot  exceed  15. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Selected  Objectives  and  Test  Items 


DESCRIPTION: 


Specification  of  the  objectives  and  test  items 
that  have  been  selected  by  an  evaluation  materials 
developer  for  inclusion  in  a  particular  knowledge 
or  performance  test  under  development  or  revision. 


DATA  SOURCE: 


DFD  2. 1.3. 2.1,  Process  1  and  Data  Store  E4 


DATA  DESTINATION:  DFD  2. 1.3. 2.1,  Data  Store  E4  and  Process  1,  2 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 
Objective  IDs 


Test  Item  IDs 


Description 

Each  objective  ID  for  which  the  test 
applies.  (An  Objective  ID  is  a  unique 
number  identifying  a  particular 
behavioral  objective  automatically 
assigned  by  the  system.) 

Data  Type:  Array  (l-X  objective  IDs) , 
where  X  cannot  exceed  6. 

The  test  item  ID  that  corresponds 
to  each  test  item  that  is  incorporated 
into  the  test.  (A  test  item  ID  is 
a  unique  number  identifying  a  particular 
test  item,  automatically  assigned  by  the 
system.) 

Data  Type:  Array  (l-X  test  item  IDs), 
where  X  cannot  exceed  100. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Task  Evaluation  Parameters 


DESCRIPTION:  Off-line  information  derived  by  an  evaluation 

materials  planner  which  is  used  to  determine 
evaluation  strategies  and  resource  requirements 
for  a  given  task.  Evaluation  parameters  result 
from  an  analysis  of  task  and  behavioral  objective 
data  combined  with  operational  environment 
information  and  include: 

Task  skill  and  knowledge  requirements  which 
must  be  evaluated. 

Task  performance  literacy  requirements 
(nomenclature)  which  must  be  evaluated, 

-  Influencing  environmental  conditions  which 
must  be  considered. 

Task  performance  variance  across  work  centers 
which  must  be  estimated,  and 

-  Probability  of  change  within  work  centers. 


DATA  SOURCE:  DFD  2.1.2,  Process  1 

DATA  DESTINATION:  DFD  2.1.2,  Process  2 


STRUCTURE:  Data  elements  shall  include: 


This  data  flow  is  situationally  specific*  therefore,  no  attempt 
is  made  to  specify  its  contents. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Task  Evaluation  Strategy  Info 


DESCRIPTION:  Off-line  information  determined  by  an  evaluation 

materials  planner,  which  provides  his/her 
perception  of  the  evaluation  strategy  that  will  be 
most  efficient  and  effective  for  rating  trainees' 
ability  to  perform  a  given  task.  This  information 
shall  identify  the  initial  assessment  of  the 
evaluation  materials  that  shall  be  needed  as  well 
as  the  resource  availability  or  constraints  that 
exist.  One  or  more  of  the  following  evaluation 
strategies  may  apply  to  the  task:  over-the- 
shoulder  observation,  knowledge  testing,  product 
evaluation,  rigged  (scenario) ,  simulation,  other. 


DATA  SOURCE:  DFD  2.1,  Process  2 

DFD  2.1.2,  Process  2 


DATA  DESTINATION:  DFD  2.1,  Process  3 

DFD  2.1.3,  Process  1,  2 

DFD  2 . 1 . 3 . 1 ,  Process  1 
DFD  2. 1.3. 1.1,  Process  1,  2,  3 
DFD  2 . 1 . 3 . 2 ,  Process  1 ,  3 
DFD  2 . 1 . 3 . 2 . 1 ,  Process  1 


This  data  flow  is  situationally  specific;  therefore,  no  attempt 
is  made  to  specify  its  contents. 


Specification  Number- 70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Task/Subtask  Performance  Resources 


DESCRIPTION:  Tools,  equipment  and  materials  identified  in  the 

Master  Task  List  (MTL)  as  required  for  successful 
performance  of  a  given  task/subtask.  These 
resources  are  used  by  a  behavioral  objective 
developer/reviser  to  specify  those  performance 
resources  which  apply  to  a  behavioral  objective 
under  development/revision  for  the  task/subtask. 


DATA  SOURCE: 


External  Entity  1.1 


DATA  DESTINATION:  DFD  2. 1.1.1,  Process  4 


STRUCTURE:  Data  elements  shall  include: 

Data  Item  Description 

Performance  Resources  List  of  tools,  equipment  and  materials 

that  were  identified  in  the  MTL  for 
the  task/ subtask. 

Data  Type:  Array  (1-X  names  and 
descriptions),  where  X  cannot  exceed  30. 


Specification  Number  70S647300B 
Code  Ident.  No,  76301 
Date:  31  October  1789 


DATA  FLOW:  Task/Subtask  References 


DESCRIPTION:  Publications  identified  in  the  Master  Task  List 

(MTL)  as  providing  task  performance  and 
proficiency  requirements  for  a  given  task/subtask. 
These  publications  are  used  by  a  behavioral 
objective  developer/reviser  to  specify  those 
references  (publications)  which  apply  to  a 
behavioral  objective  under  development,/ revision 
for  the  task/subtask. 


DATA  SOURCE:  External  Entity  1.1 


DATA  DESTINATION:  DFD  2. 1.1.1,  Process  3 


STRUCTURE:  Data  elements  shall  include: 


References 


List  of  references  (publications)  that 
were  identified  in  the  MTL  for  the  task/ 
subtask.  Each  reference  is  identified 
by  title  and  identification  code. 

Data  Type:  Array  (1-X  titles  and  IDs), 
where  X  cannot  exceed  6. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Task/Subtask  Statement 


DESCRIPTION:  Statement  which  adequately  and  accurately 

describes  the  performance  or  action  required  to 
■  accomplish  a  given  task  or  subtask.  The  statement 
contains  an  object  and  an  action  verb.  The 
statement  is  used  by  a  behavioral  objective 
developer  to  develop  one  or  more  BO  statements 
for  the  task/subtask.  The  statement  may  be 
identical  to  an  OSR  task  statement,  a  revision  of 
an  OSR  task  statement,  or  a  statement  created  by  a 
Subject  Matter  Expert  (SME) . 


DATA  SOURCE:  External  Entity  1.1 


DATA  DESTINATION:  DFD  2. 1.1.1,  Process  1  and  2 


STRUCTURE:  N/A.  Data  Type:  Alpha/numeric  string,  not 

to  exceed  240  characters/spaces. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Test  Analysis  Data 


DESCRIPTION:  Data  applicable  to  a  given  test,  which  are  used 

to  analyze  the  test  and  it's  test  items.  The 
data  are  extracted  from  data  stored  for  the  test, 
and  are  consolidated  with  test  item  analysis 
and  test  score  data  to  perform  an  analysis  of 
the  test/test  items. 


DATA  SOURCE: 


Data  Store  E4 


DATA  DESTINATION: 


DFD  2. 1.3. 2,  Process  2 
DFD  2. 1.3. 2. 2,  Process  1 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 
Test  ID  Data 

Selected  Objectives 
and  Test  Items 


Test  Format  Data 
-  Order  of  Items 


-  Critical  Items 

-  Pass  Criterion 


Test  Parameter  Data 
for  Analysis 


Description 

Includes  task  ID,  subtask  ID  (if 
applicable) ,  objective  ID,  and  test  ID. 

Behavioral  objectives  and  test  items 
selected  for  inclusion  in  the  test. 
Behavioral  objectives  are  identified  by 
BO  IDs;  test  items  are  identified  by 
BO  ID  and  Test  Item  ID. 


Order  in  which  the  items  will  appear  in 
the  test  at  presentation  (if  the  item 
scrambling  option  in  the  test  parameters 
has  not  been  invoked) . 

Items  designated  as  critical  items. 

(Knowledge  test  only)  Number  of  correct 
responses  required  to  achieve  a  passing 
score  on  the  objective. 

Includes  the  following  parameters:  item 
analysis  enabled  and  item  scrambling 
enabled  or  disabled. 


147 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Test  Analysis  Request  Data 


DESCRIPTION:  An  on-line  request  for  an  analysis  of  a  given  test 

and  its  test  items.  The  data  are:  entered  via 
the  keyboard  by  a  test  analyzer;  and  used  by  the 
system  to  identify  and  extract  the  respective  test 
samples  and  test  items  to  be  analyzed. 


DATA  SOURCE: 


DFD  2 . 1 . 3 . 2 . 2 ,  Process  2 


DATA  DESTINATION:  DFD  2. 1.3. 2. 2,  Process  1 


STRUCTURE:  Data  elements  will  include: 


Data  Item 

Test  Key 

Identification 

Data 


Test  Analysis 
Selection  Criteria 


Description 

Identification  data  which  are  keyed  to 
permit  an  analysis  of  a  given  test  to  be 
performed.  Such  data  are  documented  for 
all  tests  on  a  product  titled:  Test  I  tarn 
Analysis  Keys;  the  applicable  keyed  data 
is  taken  from  the  product  and  entered 
into  the  keyboard  by  a  test  analyzer. 

Data  specifying  the  criteria  for  analysis, 
to  include  one  or  more  of  the  following: 

-  Test  Media  (on-line  samples,  off-line 
samples  or  all  samples) 

-  Samples  of  the  test 
administered/scored  during 

a  specified  timeframe  (data  for 
begin  date  and  end  date  will  also 
apply) , 

Samples  of  the  test  which  have 
been  failed, 

Samples  of  the  test  which  have 
been  passed,  or 

-  All  samples  of  the  test. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 

Page  1  of  2 


DATA  FLOW:  Test  Data 


DESCRIPTION:  Complete  specifications  developed  or  revised  for  a 

particular  performance  or  knowledge  test. 


DATA  SOURCE:  DFD  2.1,  Process  3  and  Data  Store  E4 

DFD  2.1.3,  Process  2  and  Data  Store  E4 
DFD  2. 1.3. 2,  Process  1  and  Data  Store  E4 


DATA  DESTINATION:  DFD  2.1,  Data  Store  E4  and  Process  3 

DFD  2.1.3,  Data  Store  E4  and  Process  2 

DFD  2. 1.3. 2,  Data  Store  E4  and  Process  1,  3 


STRUCTURE:  Data  elements  shall  include: 


Selected  Objectives 
and  Test  Items 


Includes  task  ID,  subtask  ID  (if 
applicable) ,  objective  ID,  and  test  ID. 


Behavioral  objectives  and  test  items 
selected  for  inclusion  in  the  test. 
Behavioral  objectives  are  identified  by 
BO  IDs;  test  items  are  identified  by 
BO  ID  and  Test  Item  ID. 


Test  Format  Data 

-  Order  of  Items  Order  in  which  the  items  will  appear  in 

the  test  at  presentation  (if  the  item 
scrambling  option  in  the  test  parameters 
has  not  been  invoked) . 


-  Critical  Items  Items  designated  as  critical  items. 


Pass  Criterion  (Knowledge  test  only)  Number  of  correct 

responses  required  to  achieve  a  passing 
score  on  the  objective. 
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Page  2  of  2 


DATA  FLOW:  Test  Data  (continued) 


STRUCTURE : 


(continued) 


Data  Item 


Description 


Test  Parameter  Data 
-  Test  Use 


-  Item  Analysis 


Maximum  Analysis 
Samples 


-  Time  Allowed 


-  Item  Scrambling 


Designation  that  the  test  is  either  the 
primary  test  of  its  type  (performance  or 
knowledge)  or  an  alternate  test. 

Item  analysis  capability  enabled  or 
disabled.  Default  shall  be  enabled. 

Maximum  number  of  samples  of  test 
scores  that  will  stored  by  the  system 
for  use  in  item  analysis.  -Range  shall 
be  30  to  100;  default  shall  be  50. 

(Knowledge  tests  only)  Time  allowed  the 
airman  for  completing  the  test.  Range 
shall  be  1  to  1440  minutes;  default 
shall  be  60  minutes. 

(Knowledge  tests  only)  Enable  or 
disable  scrambling  of  items  at 
presentation.  Default  shall  be 
disabled. 


-  Test  Interruption 


-  Test  Item  Recap 


Test  Instructions 


(Knowledge  tests  only)  Interruption 
to  be  allowed/disallowed  during  testing. 
Default  shall  be  allowed. 

(Knowledge  tests  only)  Recapitulation 
of  missed  test  items  with  correct 
response  displayed  to  be 
allowed/disallowed  during  online 
testing.  Default  shall  be  disallowed. 

(Knowledge  test  only) .  Instructions 
which  include  the  test  purpose  and 
testing  procedures. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Test  Data  for  Event  Scheduling 


Data  applying  to  a  particular  test,  which  is  passed 
to  the  Management  Subsystem  to  enable  the 
scheduling  of  an  evaluation  event.  These  data 
shall  support  either  a  task  performance  evaluation 
or  a  Quality  Control  (QC)  evaluation  event. 


DFD  2.1,  Data  Store  E4 
DFD  2.1.3,  Data  Store  E4 
DFD  2. 1.3. 2,  Data  Store  E4 


DATA  DESTINATION:  External  Entity  1.3 


STRUCTURE:  Data  elements  shall  include: 

Data  Items 
Test  ID 


Administration 

Data 


Descriptions 

The  unique  number  assigned  by  the  system  to 
identify  the  test  among  other  tests  developed 
for  a  given  task/subtask/objective. 

The  designation  of  whether  the  test  can  be 
administered  on  line,  off  line  or  either, 
depending  on  the  developer's  instructions. 
(Only  applies  if  test  is  a  knowledge  test; 
a  performance  test  is  always  administered 
off  line.) 


DATA  FLOW: 

DESCRIPTION: 

DATA  SOURCE: 
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Specification' Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Test  Format  Data 


DESCRIPTION:  Data  for  a  particular  test  under  development  or 

revision  that  identifies  critical  test  items  and 
establishes  the  order  in  which  the  test  items 
shall  be  presented. 


DATA  SOURCE:  DFD  2. 1.3. 2.1,  Process  2  and  Data  Store  E4 


DATA  DESTINATION:  DFD  2. 1.3. 2.1,  Data  Store  E4  and  Process  2 


STRUCTURE:  Data  elements  shall  include: 

Data  Item  Description 

Set  Critical  A  test  item  that  is  considered  critical 

to  task  performance  shall  be  flagged  as 
critical.  All  critical  items  must  be 
passed. 

Test  Item  Order 


The  order  in  which  the  test  items  shall 
be  presented  to  the  airman/ trainee  who 
is  being  evaluated. 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Test  ID 


DESCRIPTION:  .  A  number  assigned  by  the  system  to  a  particular 
test  that  was  developed  for  a  particular 
objective.  The  Test  ID  shall  be  used  to  key 
(link) : 


The  test  format,  parameters,  and  knowledge 
testing  instructions  (if  applicable)  and 


- 

The 

test  to  each  test  item  incorporated  in 

the 

test. 

DATA 

SOURCE: 

DFD 

2.1.3, 

Process  2 

DFD 

2. 1.3. 2, 

Process  1 

DFD 

2. 1.3. 2.1, 

Process  1 

DATA 

DESTINATION: 

DFD 

2.1.3, 

Process  1 

DFD 

2. 1.3.1, 

Process  3 

DFD 

2. 1.3. 2.1, 

Data  Store  E4 

STRUCTURE 


N/A.  Data  Type:  Integer  (number  is  limited 
only  by  system's  capacity  for  relative 
records  per  file) . 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Test  Item 


DESCRIPTION:  An  oral  test  guide  (OTG) ,  a  performance  evaluation 

checklist  (PEC) ,  or  any  one  of  the  eight  types  of 
knowledge  test  items  (true/ false,  multiple  choice, 
list  multiple  choice,  matching,  constructed 
response,  limited  constructed  response,  single 
area  touch,  or  sequence  touch)  under  development 
or  revision  for  a  given  behavioral  objective. 


DATA  SOURCE:  DFD  2. 1.3.1,  Process  1  and  Data  Store  E3 


DATA  DESTINATION:  DFD  2. 1.3.1,  Data  Store  E3  and  Process  1 


STRUCTURE:  Data  elements  shall  include: 

Data , Item  Description 

Refer  to  Data  Flows  titled:  OTG,  PEC,  or  Knowledge  Test  Item. 
Data  items/ descript ions  applicable  to  these  data  flows  also  apply 
to  this  data  flow. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  Test  Item  Analysis  Data 


DESCRIPTION:  Data  which  are  used  to  analyze  the  test  items  for 

a  specific  test.  The  data  are  extracted  from  the 
Test  Item  Bank  and  consolidated  with  test 
analysis  and  test  score  data  for  analyzing  the 
test  and  each  of  its  items. 


DATA  SOURCE: 


Data  Store  E3 


DATA  DESTINATION:  DFD  2. 1.3. 2,  Process  2 

DFD  2. 1.3. 2. 2,  Process  1 


STRUCTURE:  Data  elements  include: 


Data  Item 
Objective  IDs 


Test  Item  IDs 


Correct  Answers 


Description 

The  behavioral  objective  ID(s)  for  which 
each  test  item  applies.  Each  BO  ID  is 
a  unique  number  assigned  by  the  system 
when  the  respective  behavior  objective 
was  developed. 

The  test  item  ID  for  each  respective 
test  item  incorporated  in  the  test. 

Each  test  item  ID  is  a  unique  number 
assigned  by  the  system  when  the 
respective  test  item  was  developed. 

The  response  alternative (s)  designated 
for  each  test  item  incorporated  in  the 
test.  Each  correct  answer  is  designated 
by  the  test  item  developer/reviser  when 
the  respective  test  item  was  developed 
or  revised. 


Alternative  Scrambling  The  designation  that  response 
Option  alternatives  were  presented  in  fixed 

order  or  were  scrambled. 
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Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  1  of  2 


DATA  FLOW:  Test  Item  Analysis  Key 


DESCRIPTION:  A  printed  product  which  identifies  test  i.eys 

for  each  test  being  stored  within  the  aOTS. 

The  listing  is  sorted  by  AFS  and  AOTS  Task  ID,  and 
is  used  by  a  test  analyzer  to  input  specific  data 
to  initiate  an  analysis  for  any  test  listed. 


DATA  SOURCE: 


DATA  DESTINATION: 


STRUCTURE : 


DFD 

2.1, 

Process  3 

and 

Data 

Store 

E7 

DFD 

2.1.3, 

Process  2 

and 

Data 

Store 

E7 

DFD 

2. 1.3. 2, 

Process  2 

and 

Data 

Store 

E7 

DFD 

2. 1.3. 2. 2, 

Process  1 

DFD 

2.1, 

Data 

Store 

E7 

and 

Process 

3 

DFD 

2.1.3, 

Data 

Store 

E7 

and 

Process 

2 

DFD 

2. 1.3. 2, 

Data 

Store 

E7 

and 

Process 

2 

DFD 

2. 1.3. 2. 2, 

Data 

Store 

E7 

and 

Process 

2 

elements  shall 

include : 

Data  Item 


Beasripti9n 


AFSC 

AOTS  Task  IDs 


Subtask  IDs 


Objective  IDs 


Alpha/numeric  code  and  title  identifying 
an  Air  Force  Specialty. 

The  task  IDs  which  apply  to  tasks  for 
which  one  or  more  tests  exist  for 
analysis.  Each  task  ID  is  an  alpha/ 
numeric  string  of  6  characters. 

The  subtask  numbers  which  apply  to 
subtasks  for  which  one  or  more  tests 
exist  for  analysis.  Each  subtask  ID 
is  a  numeric,  not  to  exceed  25. 

The  behavioral  objective  IDs  which  apply 
to  objectives  for  which  one  or  more 
tests  exist  for  analysis.  Each 
Objective  ID  is  a  numeric. 
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DATA  FLOW: 

STRUCTURE : 
Data  Items 
Test  IDs 

Test  Keys 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Test  Item  Analysis  Key  (continued) 


(continued) 


Descriptions 

The  test  IDs  which  apply  to  tests 
existing  for  analysis.  Each  test  ID 
is  a  numeric. 

The  keys  which  apply  to  tests  existing 
for  analysis.  Each  test  key  is  a 
sequential  array  of  six  numeric,  one  or 
two  digit,  codes  that  must  be  input  via 
the  keyboard  to  perform  an  analysis  of 
the  respective  test  and  its  test  items. 
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DATA  FLOW:  Test  Item  Data 


DESCRIPTION:  All  data  developed  or  revised  for  a  particular 

test  item,  to  include:  identification  data,  text, 
specifications,  and  references. 

For  the  Evaluation  Instrument  Management 
Component:  these  data  are  maintained  with  other 
test  item's  data  within  the  Test  Item  Bank;  are 
used  when  developing/revising  a  test  which 
incorporates  the  test  item;  and  are  used  when 
validating  the  test  item. 


DATA  SOURCE:  DFD  2.1,  Process  3  and  Data  Store  E3 

DFD  2.1.3,  Process  1  and  Data  Store  E3 

DFD  2. 1.3. 2,  Data  Store  El 
DFD  2. 1.3. 2.1,  Data  Store  El 


DATA  DESTINATION:  DFD  2.1,  Data  Store  E3  and  Process  3 

DFD  2.1.3,  Data  Store  E3  and  Process  1 ,  2 

DFD  2 . 1 . 3 . 2 ,  Process  1 ,  2 

DFD  2 . 1 . 3 . 2 . 1 ,  Process  1 


STRUCTURE:  Data  elements  include: 


Data 


Test  Item 

Identification 

Data 


Data  that  identifies  a  particular  test 
item,  including  task/subtask  ID, 
behavioral  objective  ID  and  test  item 
ID. 


Test  Item  Type  Oral  test  guide  (OTG) ,  performance 

evaluation  checklist  (PEC) ,  or  any  of 
the  eight  types  of  knowledge  test  items: 
true/false,  multiple  choice,  list 
multiple  choice,  matching,  constructed 
response,  limited  constructed  response, 
single  area  touch,  or  sequence  touch. 
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DATA  FLOW:  Test  Item  Data  (continued) 


STRUCTURE : 
Data  Item 


(continued) 

Description 


Test  Item  Contents 


Test  Item  References 
and  Breakdowns 


Test  IDs 


Test  item  text  and  specifications.  The 
contents  depend  on  the  type  of  test 
item.  Refer  to  the  data  flows  titled: 
OTG,  PEC  or  Knowledge  Test  Item,  for 
descriptions  of  test  item  contents  data. 


Titles,  identification  codes,  and 
breakdowns  of  the  publications  that 
specify  performance  and  proficiency 
requirements  applicable  to  the  test 
item. 

The  unique  numerical  test  IDs  for  tests 
which  incorporate  the  test  item. 
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DATA  FLOW:  Test  Item  Evaluation  (TIE)  Report 


DESCRIPTION:  A  report  which  documents  test,  subscale 

(objective)  and  test  item  results  for  a  given 
test.  This  report  is  generated  once  an  automated 
analysis  of  the  test  and  its  test  items  are 
performed  for  the  purpose  of  validating  the 
evaluation  materials. 


DATA  SOURCE: 


DFD  2. 1.3. 2,  Process  2 
DFD  2. 1.3. 2. 2,  Process  2 


DATA  DESTINATION :  DFD  2 . 1 . 3 . 2 ,  Process  3 


STRUCTURE : 


Data  elements  shall  include: 


Data  Item 


Description 


Analyzed 

Test  Data  Number  of  samples  included  in  the 

analysis;  date  range  of  the  included 
samples;  number  of  test  failures  due 
to  total  percent;  number  of  test 
failures  due  to  critical  objective 
failed  (every  objective  shall  be 
considered  critical) . 


Analyzed  Subscale  Number  of  failures;  mean  score;  standard 

(Objective)  Data  deviation;  critical  objective  (yes/no) ; 

number  of  items  associated  with 
objective;  number  of  items  needed  to 
pass  objective;  alpha  reliability;  item 
correlation. 


Analyzed  Item  Data  Subscale  associated  with  item;  mean; 

(for  each  test  item)  standard  deviation;  number  of  failures; 

critical  item  (yes/no) ;  test  item 
correlation  to  objective;  test  item 
correlation  to  total  test;  correct 
alternative ( s) ;  alternative (s)  raw 
scores;  alternative (s)  percent  score. 
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DATA  FLOW:  Test  Item  ID 


DESCRIPTION:  A  unique  numeric  identifier  assigned  by  the  system 

to  a  given  test  item.  A  test  item  ID  shall  be: 

Stored  with  other  data  developed/revised  for 

the  test  item  in  the  Test  Item  Bank, 

to  key  (link)  the  respective  test  item  data, 

Passed  to  the  Management  Component,  to  key 
(link)  the  test  item  to  the  respective 
references . 


DATA  SOURCE:  DFD  2.1,  Process  3 

DFD  2.1.3,  Process  1 

DFD  2. 1.3.1,  Process  1 
DFD  2. 1.3. 1.1,  Process  1,  2,  3 


DATA  DESTINATION:  DFD  2.1,  External  Entity  1.1 

DFD  2.1.3,  External  Entity  l . l 

DFD  2. 1.3.1,  External  Entity  1.1  and 
Data  Store  E3 

DFD  2. 1.3. 1.1,  External  Entity  l.l  and 
Data  Store  E3 


STRUCTURE:  N/A.  Data  Type:  Integer  (number  is  limited 

only  by  system's  capacity  for  relative 
records  per  file) . 
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DATA  FLOW:  Test  Item  References 


DESCRIPTION:  Data  identifying  the  publications  and  their 

breakdowns  which  specify  the  performance  and 
proficiency  requirements  which  apply  to  a 
given  test  item  under  development/revision. 

Test  Item  references  are  selected  from  references 
identifed  for  the  behavioral  objective  for  which 
the  test  item  applies.  Test  Item  publication 
breakdowns  are  itemized  for  each  selected 
publication . 


DATA  SOURCE:  DFD  2. 1.3.1,  Process  2  and  Data  Store  E3 

DATA  DESTINATION:  DFD  2. 1.3.1,  Data  Store  E3  and  Process  2 


STRUCTURE:  Data  elements  shall  include: 

Data  Item  Description 

Test  Item  Publications  listed  by  identification 

References  code  and  title  that  apply  to  the  test 

item. 

Data  Type:  Array  (l-X  publication 
IDs/titles;  where  X  cannot  exceed  6). 

Test  Item 

Publication  Breakdowns  A  list  of  particular  segments  (chapters, 

paragraphs,  tables,  etc.)  for  each 
test  item  reference. 

Data  Type:  Array  (l-X  breakdowns;  where 
X  cannot  exceed  100) . 
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DATA  FLOW:  Test  Item  Revision  Requirement (s) 


DESCRIPTION:  A  test  validator's  perception  of  changes  that  are 

needed  in  a  particular  test  in  order  to  make  the 
test  more  effective  for  evaluating  task  knowledge 
and/or  performance.  For  example:  a  performance 
evaluation  checklist  (PEC)  might  need  to  be 
revised  to  include  additional  steps  or  to  clarify 
misleading  or  inaccurate  steps;  a  knowledge  test 
item  might  need  to  be  revised  because  one  or  more 
of  the  alternatives  do  not  adequately  discriminate 
between  individuals  who  have  the  requisite 
knowledge  and  those  who  do  not. 

Based  on  the  test  item  revision  requirement (s) , 
the  test  item  developer  may  need  to  develop  one  or 
more  new  items,  revise  existing  test  items  or 
delete  inadequate  test  items. 


DATA  SOURCE: 


DFD  2.1.3,  Process  2 
DFD  2. 1.3. 2,  Process  3 


DATA  DESTINATION:  DFD  2.1.3,  Process  1 

DFD  2. 1.3.1,  Process  1 

DFD  2. 1.3. 1.1,  Process  1,  2,  3 


STRUCTURE:  Data  elements  include: 

Data  Item  Description 

This  data  flow  is  situationally  specific;  therefore,  no  attempt 
was  made  to  specify  its  contents. 
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DATA  FLOW:  Test  Parameter  Data 


DESCRIPTION:  Parameters  that  have  been  established  for  a 

particular  performance  or  knowledge  test  by  the 
test  developer/reviser.  These  data  shall  be 
stored  with  other  data  being  maintained  for  the 
test;  and  are  revised  as  deemed  necessary. 


DATA  SOURCE:  DFD  2. 1.3. 2.1,  Process  3  and  Data  Store  E4 


DATA  DESTINATION:  DFD  2. 1.3. 2.1,  Data  Store  E4  and  Process  3 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 
Time  Allowed 


Test  Use 


Item  Analysis 


Description 

Maximum  time  limit  for  completing  the 
test.  Default  shall  be  60  minutes; 
range  shall  be  1  to  1440  minutes. 

Test  may  be  designated  as  the  primary 
or  an  alternate  test  for  its  test 
category  (performance  or  knowledge) . 

Item  analysis  may  be  enabled  or 
disabled. 


Maximum  Analysis 
Samples 


Item  Scrambling 


Number  of  scored  samples  of  the  test 
that  the  system  will  accumulate  and 
store  for  analysis/validation  purposes. 
When  the  set  number  has  been 
reached,  no  more  scored  samples  of  the 
test  will  be  retained.  Default  shall  be 
50;  the  range  is  30  to  100. 

Items  may  be  presented  in  fixed  or 
scrambled  order.  Scrambled  order  will 
result  in  the  items  appearing  in  a 
different  order  each  time  the  test  is 
generated  for  online  use  or  is  printed. 
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Page  2  of  2 

DATA  FLOW:  Test  Parameter  Data  (continued) 


STRUCTURE:  (continued) 


Data  Item 


Description 


Test  Interruption 


Test  Recapitulation 


The  test  developer  may  allow  or  disallow 
the  interruption  of  a  knowledge  test 
administered  on  line. 

The  test  developer  may  allow  or  disallow 
an  airman  taking  the  test  on  line  to 
view  answered  test  items  and  the 
respective  correct  answers  and  feedback. 
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DATA  FLOW:  Test  Score  Data 


DESCRIPTION:  Data  which  result  from  the  scoring  of  a  given 

knowledge  or  performance  test  (see  Process  2.2.3 
of  the  Performance  Evaluation  Component) .  These 
data  are  aggregated  for  each  scored  sample  of  the 
test  (up  to  the  maximum  number  of  samples 
designated)  and  are  stored  with  test  score  data 
for  other  scored  tests  (see  Data  Store  E6) . 

Within  the  Evaluation  Instrument  Management 
Component,  the  aggregated  data  are  extracted  and 
used,  along  with  other  data,  to  analyze  the  test 
and  its  test  items  for  the  purpose  of  validating 
the  evaluation  materials.  Some  test  score  data 
applies  to  either  a  performance  or  a  knowledge 
test;  other  test  score  data  depends  on  the  type  of 
test  scored. 


DATA  SOURCE:  Data  Store  E6 


DATA  DESTINATION:  DFD  2.1,  Process  3 

DFD  2.1.3,  Process  2 
DFD  2 . 1 . 3 . 2 ,  Process  2 
DFD  2 . 1 . 3 . 2 . 2 ,  Process  1 


STRUCTURE:  Data  elements  shall  include: 


Test  Identification 
Data 


Data  identifying  the  specific  test,  to 
include  the  task  ID,  subtask  ID  (if 
applicable) ,  behavioral  objective  ID  and 
test  ID. 


Dates  of  Administration  Range  of  dates  during  which  the  test 

samples  were  scored. 


Number  of  Samples 


Number  of  test  samples  scored. 
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Page  2  of  2 
DATA  FLOW: 
STRUCTURE : 


Test  Score  Data  (continued) 
(continued) 


Test  Item  Data  identifying  each  test  item 

Identification  Data  incorporated  in  the  test,  to  include  the 

behavioral  objective  Id  and  test  item 
ID. 


Performance  Test  Score  Data 

-  Pass/Fail  Data  Total  number  of  test  failures. 

-  Step  Pass/Fail  Number  of  steps  failed  by  each  examinee, 

Data  specific  steps  failed,  number  of  steps 

unobserved,  specific  steps  unobserved. 

-  Critical  step  Critical  steps  failed,  number  of  times 

each  critical  step  was  failed. 


Knowledge  Test  Score  Data 

-  Pass/Fail  Data  Total  number  of  test  failures. 

-  Percentage  Percentage  of  correct  items  on  each 

Correct  test  administration. 

-  Incorrect  Items  Identification  of  test  items  failed  and 

number  of  failures  on  each  item. 

-  Response  Response  alternatives  selected  on  each 

Alternative  item  and  number  of  times  each  response 

Data  alternative  was  selected. 

-  Objectives  Identification  of  failed  objectives  and 

Failed  and  number  of  times  each  objective  was 

failed. 

-  Critical  Items  Identification  of  failed  critical  items 

and  number  of  times  that  each  was 
failed. 
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DATA  FLOW:  Test  Revision  Requirement (s) 


DESCRIPTION:  A  test  validator's  perception  of  changes  that  are 

needed  in  a  particular  test  in  order  to  make  it  a 
more  effective  instrument  for  task  performance 
and/or  knowledge  evaluation.  Based  upon  the  test 
revision  requirement (s) ,  the  test  developer  might 
need  to  develop  new  items  for  the  test,  delete 
items  from  the  test,  develop  a  new  test,  delete  an 
inadequate  test,  reformat  the  test,  redefine  the 
test  parameters,  or  make  some  other  revision. 


DATA  SOURCE:  DFD  2. 1.3. 2,  Process  3 


DATA  DESTINATION:  DFD  2. 1.3. 2,  Process  1 


STRUCTURE : 


Data  elements  shall  include: 


This  data  flow  is  situationally  specific;  therefore,  no  attempt 
is  made  to  specify  its  contents. 
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DATA  FLOW:  Training/Evaluator  Resource  Data  for  Inventory 


DESCRIPTION:  Specification  of  resources  (tools,  equipment  and 

materials)  required  for  training  and/or 
evaluation  for  a  given  task  which  are  not 
already  listed  in  the  AOTS  inventory  of  resources. 
These  resources  are  defined  by  a  behavioral 
objective  developer/reviser  when  specifying  the 
resources  for  a  particular  behavioral  objective. 
The  name  of  each  resource  is  passed  to  the 
Management  Subsystem,  whereby  additional  data  for 
the  resource  is  then  updated  to  the  AOTS 
inventory . 


DATA  SOURCE 


DFD  2.1, 

DFD  2.1.1, 
DFD  2. 1.1.1 


Process  1 
Process  1,  2 
Process  4 


DATA  DESTINATION:  External  Entity  1.3 


STRUCTURE:  Data  elements  include: 


Data  Item 

Training  Resources 


Evaluator  Resources 


.P3.s<?riPti9n 

Name  of  each  training  resource  not  found 
in  the  AOTS  inventory  of  resources. 

Data  Type:  Alpha/ numeric  string,  not  to 
exceed  40  characters/spaces. 

Name  of  each  evaluator  resource  not 
found  in  the  AOTS  inventory  of 
resources . 

Data  Type:  Alpha/numeric  string,  not  to 
exceed  40  characters/spaces. 
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Evaluation  Instrument  Management  Component  Data  Stores. 


20.2.2 


Page  1  of  3 
DATA  STORE: 

DESCRIPTION: 


DATA  SOURCE: 

STRUCTURE : 

Data  Item 
Objective  ID 
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El,  Consolidated  BO  Data 


Data  stored  for  all  behavioral  objectives  which 
have  been  developed  or  revised  for  tasks  or 
subtasks  for  all  AFSs  operating  under  the  AOTS. 

As  data  for  each  objective  are  developed,  the 
data  are  stored  within  this  central  data 
file;  as  data  are  revised,  the  existing  data 
are  extracted  from  the  file  and  replaced  by 
the  revised  data. 

Objective  identification  data,  the  complete 
behavioral  objective  statement,  references 
and  their  breakdowns,  resources  and  training 
materials  applicable  to  each  behavioral 
objective  shall  be  stored  in  this  file. 

-  Once  a  behavioral  objective  is  deleted, 

all  data  for  that  BO  shall  be  automatically 
deleted  from  this  datastore. 


DFD  2.1,  Process  1 

DFD  2.1.1,  Process  1 ,  2 

DFD  2. 1.1.1,  Process  2,  3,  4,  5 


An  online  file  which  shall  include  the  following 
data  items  for  gach  objective: 

Description 

A  unique  number  identifying  the 
particular  behavioral  objective  for 
which  the  data  applies.  This  ID  is 
automatically  assigned  by  the  system 
once  the  BO  statement  is  developed,  and 
cannot  be  revised. 

Data  Type:  Integer  (number  is  limited 
only  by  system's  capacity  for  relative 
records  per  file) . 
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DATA  STORE:  El,  Consolidated  BO  Data  (continued) 


STRUTURE:  (continued) 


Data  Items 


Descriptions 


Complete  Behavioral  Objective  Statement 


Behavioral  A  brief  statement  headed  by  an  active 

Component  verb  that  specifies  the  required 

behavior. 


-  Conditions 
Component 


The  operational  conditions  under  which 
the  required  behavior  will  normally 
occur. 


-  Standards 
Component 


BO  References  Data 
-  Publications 


Breakdowns 


A  measurable  minimum  level  of 
performance  that  shall  be  considered  to 
demonstrate  an  acceptable  level  of 
competency. 

Data  Type  for  complete  statement: 
alpha/ numeric  string,  not  to  exceed  240 
characters/ spaces . 


Publications  listed  by  identification 
code  and  and  title  that  define  the 
performance  and  proficiency 
requirements  for  the  objective. 

Data  Type:  Array  (1-X  publication 
IDs/titles;  where  X  cannot  exceed  6) . 

Breakdowns  (by  section,  chapter,  figure, 
table,  page,  or  other  part)  of  the 
publications  that  define  the  performance 
and  proficiency  requirements  for  the 
objective. 

Data  Type:  Array  (l-X  breakdowns;  where 
X  cannot  exceed  100) . 
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DATA  STORE:  El,  Consolidated  BO  Data  (continued) 


STRUCTURE:  (continued) 


Data  Items 
BO  Resources 


BO  Training  Materials 


Descriptions 

List  of  tools,  equipment  and  material 
required  for  successful  performance  of 
the  objective  (called  performance 
resources) ,  as  well  as  the  training  and 
evaluator  resources  required  to 
accomplish  the  objective. 

Data  Type:  Array  (1-X  performance 

resources) ;  Array  (l-X  evaluator 

resources) ;  Array  (1-X  training 

resources) ;  where  X  cannot  exceed  30. 

List  of  training  materials  which  exist 
for  accomplishing  the  objective.  These 
materials  can  be  of  many  types:  Computer 
Assisted  Instruction  (CAI)  lessons, 
films,  sound-on-slide  present ions,  etc. 
Data  for  each  material  includes:  Name 
of  item,  type  of  item  and  other 
identification  data  (course  ID,  film 
number/ID,  etc.) 

Data  Type:  Array  (1-X  materials) ,  where 
X  cannot  exceed  15. 
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DATA  STORE: 

DESCRIPTION: 

DATA  SOURCE: 

STRUCTURE : 

Data  Item 
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E2 ,  Bad  Verb  List 


A  list  of  inactive  verbs  maintained  on  line  by 
behavioral  objective  developers  and  revisers. 
Verbs  maintained  on  this  list  are  matched  to 
the  verb  included  in  each  behavioral  objective 
statement  under  development/revision;  a  match 
requires  the  BO  developer/reviser  to  change  the 
BO  statement  verb. 


DFD  2.1,  Process  1 
DFD  2.1.1,  Process  3 


A  file  which  shall  maintain  up  to  100  bad  verbs. 
Data  elements  shall  include: 

Description 

Each  inactive  verb  not  acceptable  for 
use  as  the  main  verb  in  a  BO  statement. 
Data  Type:  Alpha  string,  not  to  exceed 
20  characters/ spaces. 


Bad  Verbs 


Page  1  of  4 

DATA  STORE: 

DESCRIPTION: 


DATA  SOURCE: 

STRUCTURE: 

Data  Item 

Test  Item 

Identification 

Data 
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E3 ,  Test  Item  Bank 


The  central  depository  of  all  data  for  all  test 
items  (Oral  Test  Guides,  Peformance  Evaluation 
Checklists  and  knowledge  test  items)  that  have 
been  developed  or  revised. 

As  data  for  each  test  item  are  developed, 
the  data  are  stored  within  this  central 
data  store?  as  data  are  revised,  the 
existing  data  are  extracted  from  the  file 
and  replaced  by  the  revised  data. 

-  Test  item  identification  data,  the  test 
item  contents  and  references  which  apply 
to  each  test  item  are  stored  within  this 
file. 

-  Once  a  test  item  is  deleted,  all  data  for 
that  test  item  shall  automatically  be 
deleted  from  this  datastore. 


DFD  2.1,  Process  3 

DFD  2.1.3,  Process  1 

DFD  2. 1.3.1,  Process  1,  2,  3 

DFD  2. 1.3. 1.1,  Process  1,  2,  3 


An  online  file  which  shall  include  the  following 
data  for  each  test  item: 

Description 

Data  that  identifies  a  particular  test 
item,  including  the  behavioral  objective 
ID  to  which  the  test  item  applies  and 
the  test  item  ID. 
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DATA  STORE:  E3,  Test  Item  Bank  (continued) 


STRUCTURE :  ( continued) 


Data  Items 


Test  Item  Type 


Test  Item  Contents 


-  OTG  contents 

—  Behavioral 
Obj ective 
Statement 


Descriptions 

Oral  test  guide  (OTG) ,  performance 
evaluation  checklist  (PEC) ,  or  any  of 
the  eight  types  of  knowledge  test  items 
true/false,  multiple  choice,  list 
multiple  choice,  matching,  constructed 
response,  limited  constructed  response, 
single  area  touch,  or  sequence  touch. 

Test  item  text  and  specifications.  The 
contents  depend  on  the  type  of  test 
item: 


The  complete  behavioral  objective 
statement  for  which  the  OTG  applies. 


r —  Evaluator 
Info 


Evaluator 

Instructions 


Information  to  be  used  by  an  evaluator 
to  help  prepare  for  a  specific  task  or 
Quality  Control  evaluation.  Such 
information  will  include:  estimated 
time  to  conduct  the  evaluation; 
prerequisite  tasks  which  should  be 
trained  and  evaluated  before  the 
evaluation  is  conducted;  whether  the 
task  is  performed  by  one  individual  or 
by  a  team;  the  tools,  equipment  and 
materials  (TEMs)  required  by  the 
evaluatee;  the  TEMs  required  by  the 
evaluator;  and  any  conditions  under 
which  the  evaluation  must  be  conducted. 

Guidelines  for  administering  the 
performance  evaluation,  including  the 
wording  of  the  spoken  instructions  that 
the  evaluator  will  say  to  the  airman/ 
trainee. 
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DATA  STORE:  E3 ,  Test  Item  Bank  (continued) 


STRUCTURE:  (continued) 

Data  Items  Descriptions 

Test  Item  Contents  (continued) 

-  PEC  contents 

—  PEC  steps  The  text  for  each  PEC  step.  There  shall 

be  a  maximum  of  60  steps. 

-  Knowledge  Test  Item  contents 

—  Item  Stem  The  statement/ question  setting  forth  the 

situation  to  which  an  examinee  must 
respond.  The  stem  may  contain  up  to 
three  graphics. 

Data  Type:  Alpha/numeric  string,  not  to 
exceed  3600  characters/ spaces. 

—  Graphics  Data  (optional) 

Graphic  ID(s)  Alphanumeric  name(s)  or  code(s)  used  to 

call  up  the  graphic (s)  selected  by 
the  test  developer/reviser  for  use  in 
the  test  item  stem.  Up  to  three 
graphic  IDs  can  be  specified. 

Data  Type:  Alpha/ numeric  string,  not 
to  exceed  10  characters/spaces. 

Graphic (s)  The  actual  graphic (s)  selected  by  the 

developer/reviser.  Up  to  three  graphics 
can  exist  for  the  test  item  stem. 

Presentation  Specifications  defined  by  the  test  item 

Specifications  developer/ reviser  concerning  the 

placement,  scaling,  rotation  and 
coloring  of  each  selected  graphic. 

Data  Type:  X/Y  coordinates  for 
placement  data;  numeric  (1-10)  for 
scaling  factor;  numeric  (0-360  degrees) 
for  rotation  data;  and  numeric  (0-8) 
for  color  data. 
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DATA  STORE:  E3 ,  Test  Item  Bank  (continued) 


STRUCTURE:  (continued) 

Data  Items  Descriptions 

Knowledge  Test  Item  Contents  (continued) 


—  Response  Response  alternatives  to  the  situation 

Alternatives  posed  in  the  item  stem  that  are  defined 

by  the  test  item  developer/ reviser . 

Data  Type  for  alternatives  depends  on 
the  type  of  knowledge  test  item. 


—  Corrrect  Response  alternative (s)  designated  by 

Answer (s)  the  test  item  developer/ reviser  as  the 

correct  response  to  the  item  stem. 

Data  Type  for  correct  answer (s)  depends 
on  the  type  of  knowledge  test  item. 

—  Feedback  Comments  by  the  test  item  developer  or 

reviser  for  each  response  alternative. 
Data  Type:  Alpha/numeric  string,  not 
to  exceed  240  characters. 


Test  Item  References  Titles,  identification  codes,  and 
and  Breakdowns  breakdowns  of  the  publications  that 

specify  performance  and  proficiency 
requirements  applicable  to  the  test 
item. 

Test  IDs  The  unique  numerical  test  IDs  for  tests 

which  incorporate  the  test  item. 
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DATA  STORE: 

DESCRIPTION: 


DATA  SOURCE: 

STRUCTURE: 

Data  Item 
Test 

Identification 

Data 
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E4 ,  Consolidated  Test  Data 


The  central  depository  of  all  data  for  all  tests 
that  have  been  developed  or  revised. 

As  data  for  each  test  are  developed,  the 
data  shall  be  stored  within  this  file;  as 
data  are  revised,  the  existing  data  are 
extracted  from  this  file  and  replaced  by  the 
revised  data. 

Test  identification  data,  applicable 
objectives  and  test  items,  test  format  data 
and  test  parameter  data  which  apply  to  each 
test  shall  be  stored  within  this  file. 
Additionally,  for  each  knowledge  test,  the 
on-line/off-line  instructions  shall  also  be 
included  within  this  file. 

-  Once  a  test  is  deleted,  all  data  for  that 

test  shall  automatically  be  deleted  from  this 
datastore . 


DFD  2.1,  Process  3 

DFD  2.1.3,  Process  2 

DFD  2. 1.3. 2,  Process  l 

DFD  2. 1.3. 2.1,  Process  1,  2,  3,  4 


An  on-line  file  which  shall  include  the  following 
data  for  each  test: 

Description 

Data  that  identifies  a  particular  test, 
including  the  behavioral  objective  ID 
to  which  the  test  mainly  applies  and 
the  test  ID. 
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DATA  STORE:  E4 ,  Consolidated  Test  Data  (continued) 


STRUCTURE:  (continued) 


Data  Items 


Descriptions 


Selected  Objectives  Behavioral  objectives  and  test  items 

and  Test  Items  selected  for  inclusion  in  the  test. 

Each  behavioral  objective  is  identified 
by  its  respective  BO  ID;  each  test  item 
is  identified  by  its  respective  Test 
Item  ID. 


Test  Format  Data 
-  Order  of  Items 


-  Critical  Items 

-  Pass  Criterion 

Test  Parameter  Data 

-  Test  Use 

-  Item  Analysis 


Order  in  which  the  items  will  appear  in 
the  test  at  presentation  (if  the  item 
scrambling  option  in  the  test  parameters 
has  not  been  invoked) . 

Items  designated  as  critical  items. 

(Knowledge  test  only)  Number  of  correct 
responses  required  to  achieve  a  passing 
score  on  the  objective. 


Designation  that  the  test  is  either  the 
primary  test  of  its  type  (performance  or 
knowledge)  or  an  alternate  test. 

Item  analysis  capability  enabled  or 
disabled.  Default  shall  be  enabled. 


-  Maximum  Analysis 
Samples 


Time  Allowed 


Maximum  number  of  samples  of  test 
scores  that  will  stored  by  the  system 
for  use  in  item  analysis.  Range  shall 
be  30  to  100;  default  shall  be  50. 

(Knowledge  tests  only)  Time  allowed  the 
airman  for  completing  the  test.  Range 
shall  be  1  to  1440  minutes;  default 
shall  be  60  minutes. 
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DATA  STORE:  E4 ,  Consolidated  Test  Data  (continued) 


STRUCTURE:  (continued) 


Test  Parameter  Data  (continued) 


-  Item  Scrambling 


(Knowledge  tests  only)  Enable  or 
disable  scrambling  of  items  at 
presentation.  Default  shall  be 
disabled. 


-  Test  Interruption 


(Knowledge  tests  only)  interruption 
to  be  allowed/disallowed  during  testing. 
Default  shall  be  allowed. 


-  Test  Item  Recap 


(Knowledge  tests  only)  Recapitulation 
of  missed  test  items  with  correct 
response  displayed  to  be 
allowed/disallowed  during  online 
testing.  Default  shall  be  disallowed. 


Test  Instructions 


(Knowledge  test  only) .  Instructions 
which  include  the  test  purpose  and 
testing  procedures. 
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DATA  STORE: 


E6 ,  Consolidated  Test  Score  Data 


DESCRIPTION:  Data  stored  for  all  knowledge  and  performance 

tests  which  have  been  scored  using  an  AOTS 
keyboard  or  an  Optical  Mark  Reader. 

As  each  sample  of  a  test  is  scored,  data 
are  stored  within  this  central  file. 

Data  shall  be  aggregated  for  a  given  test,  as 
each  test  sample  is  scored,  up  to  the 
maximum  number  of  samples  designated  by  the 
test  developer. 

Data  shall  be  extracted  for  a  given  test  for 
the  purpose  of  analyzing  the  test  and  its 
test  items. 

DATA  SOURCE:  Performance  Evaluation  Component  (see  DFD  2.2.3) 


STRUCTURE : 


An  on-line  file  which  shall  include  the  following 
data  for  each  test: 


Test 

Identification 

Data 


Data  that  identifies  a  particular  test, 
including  the  behavioral  objective  ID 
to  which  the  test  mainly  applies  and 
the  test  ID. 


Dates  of  Administration  Range  of  dates  during  which  the  test 

samples  were  scored. 


Number  of  Samples 


Number  of  test  samples  scored. 


Test  Item 

Identification  Data 


Data  identifying  each  test  item 
incorporated  in  the  test,  to  include  the 
behavioral  objective  Id  and  test  item 
ID. 
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DATA  STORE:  E6,  Consolidated  Test  Score  Data  (continued) 


STRUCTURE:  (continued) 

Data  Items  Descriptions 

Performance  Test  Score  Data 

-  Pass/Fail  Data  Total  number  of  test  failures. 

-  Step  Pass/Fail  Number  of  steps  failed  by  each  examinee. 

Data  specific  steps  failed,  number  of  steps 

unobserved,  specific  steps  unobserved. 

-  Critical  Step  Critical  steps  failed,  number  of  times 

each  critical  step  was  failed. 


Knowledge  Test  Score  Data 


-  Pass/Fail  Data 


Total  number  of  test  failures. 


-  Percentage 
Correct 


Percentage  of  correct  items  on  each 
test  administration. 


-  Incorrect  Items 


Identification  of  test  items  failed  and 
number  of  failures  on  each  item. 


-  Response 
Alternative 
Data 


Response  alternatives  selected  on  each 
item  and  number  of  times  each  response 
alternative  was  selected. 


-  Objectives 
Failed 


Identification  of  failed  objectives  and 
and  number  of  times  each  objective  was 
failed. 


-  Critical  Items  Identification  of  failed  critical  items 

and  number  of  times  that  each  was 
failed. 
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DATA  STORE: 


E7 ,  Test  Key  File 


DESCRIPTION:  A  file  which  contains  data  from  which  the  most 

current  Test  Item  Analysis  Key  listing  is 
produced.  Data  in  the-  file  are  sorted  by 
objective  ID.  From  this  file,  a  listing  is 
produced  which  aligns  test  identification  data 
and  test  keys  to  enable  the  analyses  of  any 
test  existing  in  the  AOTS. 


DATA  SOURCE: 


DFD  2.1, 

DFD  2.1.3, 

DFD  2. 1.3. 2, 
DFD  2. 1.3. 2. 2, 


Process  3 
Process  2 
Process  2 
Process  l 


STRUCTURE: 


An  on-line  file  which  shall  contain: 


Objective  IDs 


Description 

The  behavioral  objective  IDs  which  apply 
to  objectives  for  which  one  or  more 
tests  exist  for  analysis.  Each 
Objective  ID  is  a  numeric. 


Test  IDs  The  test  IDs  which  apply  to  tests 

existing  for  analysis.  Each  test  ID 
is  a  numeric. 


The  keys  which  apply  to  tests  existing 
for  analysis.  Each  test  key  is  a 
sequential  array  of  six  numeric,  one  or 
two  digit,  codes  that  must  be  input  via 
the  keyboard  to  perform  an  analysis  of 
the  respective  test  and  its  test  items. 


Test  Keys 
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PROCESS:  2.2,  Evaluate  Performance 


LEVEL  OF  AUTOMATION:  Manual,  automated,  automatic 


DESCRIPTION:  This  process  shall  enable  the  evaluation  of 

trainees'  knowledge  and  performance  abilities 
with  regards  to  AFS  task  training  and 
certification.  This  process  involves  the 
administration  and  scoring  of  AFS  task  knowledge 
and  performance  tests;  and  shall  support 
pre-training,  post-training  and  Quality  Control 
(QC)  evaluations.  This  process  shall  also  enable 
the  control/accountability  of  offline  evaluation 
materials . 


DATA  INPUT:  Evaluation  Events  Data 

Test  Items  Data 
Tests  Data 

Evaluation  Materials  for  Accountability 


DATA  OUTPUT:  Knowledge  Test  Results  Notices 
Failed  QC  Evaluation  Notices 
Evaluation  Results  Data 


STRUCTURE:  This  process  includes  the  following  subprocesses: 

-  Process  2.2.1,  Administer  Knowledge  Test  On  line 

-  Process  2.2.2,  Administer  Evaluation  Off  line 

-  Process  2.2.3,  Score  Test  and  Provide  Feedback 

-  Process  2.2.4,  Account  for  Cff- ' ine  Evaluation 

Materials 
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PROCESS:  2.2.1,  Administer  Knowledge  Test  On  Line 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  the  administration  of  a 

knowledge  test  on  line.  On-line  administration  of 
a  test  shall  include: 

1.  Automatic  presentation  of  on-line  test 
instructions, 

2.  Automatic  presentation  of  questions  (one  at  a 
time) , 

3.  Ability  for  trainee  to  mark  one  or  more 
questions  for  review, 

4.  Ability  for  trainee  to  select  one  or  more 
answers  for  each  question  and  change  the 
answer  for  any  question, 

5.  Ability  for  trainee  to  review  all  questions 
or  only  marked  questions,  before  test  is 
automatically  scored,  and 

6.  Automatic  timing  while  the  trainee 
accomplishes  the  test  (to  preclude  a  trainee 
from  taking  longer  than  the  period  of  time 
as  determined  by  the  test  developer) . 

The  system  shall  enable  a  trainee  to  take  a 
knowledge  test  on  line,  under  the  following 
conditions : 
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DESCRIPTION: 


DATA  INPUT: 

DATA  OUTPUT: 

STRUCTURE: 
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Administer  Knowledge  Test  On  Line  (continued) 


(continued) 

The  trainee  uses  the  AOTS  to  determine 
his/her  next  AFS  task  training  requirement, 
whereby  the  system  automatically  schedules  an 
evaluation  event.  The  on-line  event  shall 
occur  for  a  task  or  subtask  which: 

—  is  identified  as  the  trainee's  next 
training  requirement, 

—  an  on-line  knowledge  test  exists  in  the 
AOTS ,  and 

—  the  trainee  has  completed  the  respective 
knowledge  training  or  is  being  evaluated 
for  pre-training  purposes  (to  determine 
the  individual's  need  for  task  training). 

The  system  shall  automatically  extract  the 
appropriate  test  and  test  item  data,  based  on 
specific  evaluation  event  data  such  as  the  Task  ID, 
Subtask  ID  (if  appropriate) ,  and  Objective  ID. 

Once  the  trainee  answers  all  questions  and 
completes  all  desired  reviews,  or  once  the  time 
period  expires  for  accomplishing  the  test,  the 
test  and  test  item  data,  and  trainee's  responses, 
shall  then  be  used  for  automatic  scoring  of  the 
test. 


Evaluation  Event  Data 
Test  Data 
Test  Item  Data 


Selected  Responses 


N/A 
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PROCESS:  2.2.2,  Administer  Evaluation  Off  Line 

LEVEL  OF  AUTOMATION:  Manual,  Automated 


DESCRIPTION:  This  process  shall  enable  an  evaluator  to 

administer  a  knowledge  or  performance  test  off 
line.  A  knowledge  test  shall  be  administered  for 
evaluating  AFS  task  knowledge;  a  performance  test 
shall  be  administered  for  evaluating  AFS  task 
performance/skills  or  for  conducting  a  Quality 
Control  (QC)  evaluation. 

For  an  off-line  knowledge  test,  this  process  shall 
involve: 

1.  Printing  the  required  number  of  copies  of 
the  test  (not  to  exceed  75  copies;  one  copy 
for  each  trainee  participating  in  the 
evaluation  event) , 

2.  Printing  the  score  key  (if  desired  for 
feedback  purposes  once  the  test  is  scored) , 

3.  Presenting  a  hard  copy  of  the  test  and  blank 
answer  sheet  to  the  appropriate  trainee (s), 

4.  Monitoring  the  testing  process,  and 

5.  Collecting  each  test  copy  and  completed 
answer  sheet  at  the  conclusion  of  the 
testing. 

For  an  off-line  performance  test,  this  process 
shall  involve: 

1.  Printing  the  required  number  of  copies  of 
the  test  (not  to  exceed  75  copies;  one  copy 
for  each  trainee  participating  in  the 
evaluation  event) , 

2.  Following  the  evaluator's  instructions 
provided  in  the  Oral  Test  Guide  (OTG) , 
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Administer  Evaluation  Off  Line  (continued) 


(continued) 

3.  Conducting  the  evaluation  for  each  person, 
i.e.,  trainee  performs  steps  or  product  is 
produced,  as  required  by  the  Performance 
Evaluation  Checklist  (PEC) ,  and 

4 .  Observing  and  recording  observations  for  each 
person,  using  (copies  of)  the  PEC. 

An  off-line  evaluation  shall  be  administered  once 
an  evaluation  event  has  been  scheduled  for  one  or 
more  trainees  for  a  given  task  or  subtask.  The 
following  conditions  shall  be  met: 

For  a  task  knowledge  evaluation  - 

1.  An  off-line  knowledge  test  exists  in  the  AOTS 
for  the  task/ subtask  and 

2.  Each  trainee  has  completed  the  respective 
knowledge  training  for  the  task/ subtask,  or 
has  been  scheduled  for  pre-training 
evaluation  to  determine  the  need  for 
training. 

For  a  task  performance  evaluation  - 

1.  A  performance  test  exists  in  the  AOTS  for  the 
task  and 

2 .  Each  trainee  has  completed  the  respective 
performance  training  for  the  task/ subtask,  or 
has  been  scheduled  for  pre-training 
evaluation  to  determine  the  need  for 
training. 
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Administer  Evaluation  Off  Line  (continued) 


(continued) 

For  a  Quality  Control  (QC)  evaluation  - 

1.  A  performance  test  exists  in  the  AOTS  for  the 
task/subtask  and 

2.  The  task,  evaluatee  and  evaluator (s)  meet 
QC  selection  criteria  (see  process  2.4.1). 


The  printing  of  off-line  evaluation  materials  shall 
be  permitted  at  the  time  the  evaluation  event  is 
scheduled,  or  anytime  thereafter,  upon  request. 

-  The  system  shall  automatically  extract  the 
appropriate  test  and  test  item  data,  based  on 
specific  evaluation  event  data  (such  as  Task 
ID,  Subtask  ID,  and  Behavioral  Objective  ID) . 

-  Test  material  accountability  data  shall  be 
automatically  maintained  as  soon  as  evaluation 
materials  are  printed  for  off-line  use. 

-  Additionally,  a  Test  Control  Number  (TCN)  shall 
be  automatically  established  and  printed  on 
each  copy  of  the  test  and  the  corresponding 
score  key  (score  key  applies  only  to  a 
knowledge  test) . 

Once  an  off-line  test  has  been  administered,  test 
data,  test  item  data  and  each  trainee's  responses/ 
evaluator's  observation  results  shall  then  be  used 
for  scoring  each  copy  of  the  test  and  updating  the 
accountability  of  the  test  materials. 


Evaluation  Event  Data 
Test  Data 
Test  Item  Data 
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PROCESS:  Administer  Evaluation  Off  Line  (continued) 


DATA  OUTPUT:  Selected  Responses 

Log  Record  Test  Data 
Knowledge  Test  Score  Key 


STRUCTURE :  N/A 
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PROCESS:  2.2.3,  Score  Test  and  Provide  Feedback 


LEVEL  OF  AUTOMATION:  Automated,  automatic,  manual 


DESCRIPTION:  This  process  shall  enable  an  on-line  or  off-line 

knowledge  or  performance  test  to  be  scored,  and 
feedback  to  be  provided  to  each  appropriate 
trainee. 

A  knowledge  test  which  is  administered  on  line 
shall  be  automatically  scored.  The  correct  answer 
for  each  knowledge  test  item  shall  be  matched  with 
the  trainee's  response.  The  items  missed  shall  be 
"flagged”,  to  enable  feedback  to  be  presented  to 
the  trainee  during  the  scoring  process.  The  total 
score  shall  be  provided  online  to  the  trainee,  as 
well  as  the  overall  test  results  of  "passed"  or 
"failed",  as  appropriate. 

-  Overall  test  results  shall  also  be  passed  to 
the  Management  Subsystem,  to  update  the 
trainee's  ATR  and  to  notify  necessary 
individuals  of  the  completion/overall  results 
of  the  evaluation. 

-  Test  scoring  data  shall  be  maintained,  to 
enable  future  analysis  and  validation  of  the 
test  and  its  test  items. 

A  knowledge  test  or  performance  test  which  is 
administered  off  line  shall  be  scored  in  one  of 
two  ways: 

1.  Reading  the  completed  answer  sheet (s)  into  an 
Optical  Mark  Reader  (OMR) ,  or 

2.  Entering  test  identification  data  and  each 
trainee's  responses/ evaluator ' s  observations 
via  the  keyboard. 
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Score  Test  and  Provide  Feedback  (continued) 


(continued) 

Answer  sheets  shall  be  developed  which  will  enable 
the  scoring  of  off-line  tests  to  occur  via  an  OMR 
or  keyboard  input.  One  type  of  answer  sheet 
shall  be  developed  for  scoring  knowledge  tests, 
while  another  type  shall  be  developed  for 
performance  tests.  An  answer  sheet  shall  contain: 

1.  Test  Identification  Blocks  (trainee  or 
evaluator  records  data  and/or  fills  in  blocks 
or  bubbles  to  represent  the  Test  Control 
Number,  trainee,  testing  date,  and  evaluator) 
and 

2.  Response  bubbles  (trainee  fills  in 

bubble (s)  which  identifies  the  correct  answer 
for  each  test  item  of  a  knowledge  test;  or 
an  evaluator  fills  in  bubbles  which 
identifies  the  observance  of  each  PEC  step 
for  a  performance  test) . 

When  a  knowledge  test  answer  sheet  is  scored  either 
by  using  an  OMR  or  by  keyboard  input,  test  results 
shall  be  automatically  printed  to  the  trainee's 
supervisor.  These  test  results  shall  identify  the 
trainee,  event,  task,  date  test  was  taken,  test 
score,  overall  pass/ fail  result  and  each  tost  item 
missed. 

-  The  overall  test  results  shall  be  passed  to 
the  Management  Subsystem,  to  update  the 
trainee's  ATR  and  to  notify  necessary 
individuals  of  the  completion  and  overall 
results  of  the  evaluation. 

In  addition,  the  supervisor  shall  use  the 
knowledge  test  score  key  (previously  printed 
to  administer  the  evaluation)  to  identify  the 
correct  answer (s)  to  each  missed  item  and  to 
provide  the  appropriate  feedback  to  the 
trainee. 
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Score  Test  and  Provide  Feedback  (continued) 


(continued) 

When  a  performance  test  answer  sheet  is  scored  by 
using  an  OMR  or  keyboard  input,  the  overall  test 
results  shall  be  passed  to  the  Management 
Subsystem,  to  update  the  trainee's  ATR  and  to 
notify  the  necessary  individuals  of  the  completion 
and  overall  results  of  the  evaluation. 

A  Quality  Control  (QC)  Failure  Notice  shall  be 
automatically  generated  to  a  trainee's  Unit 
Commander  if  the  performance  test  is 
administered  for  the  purpose  of  conducting  a 
QC  evaluation  and  the  trainee  fails  the 
evaluation.  This  notice  shall  advise  the 
commander  that  the  trainee  failed  the  QC 
evaluation  and  is  recommended  for 
decertification  and  remedial  training  before 
recertification. 


When  a  knowledge  or  performance  test  answer  sheet 
is  scored  by  using  an  OMR  or  by  keyboard  input, 
test  score  data  shall  be  automatically  maintained, 
to  enable  future  analysis  and  validation  of  the 
test  and  its  test  items. 

-  Overall  test  results  (passed/ failed)  shall  also 
be  used  for  automatic  accountability  of  the 
offline  materials  (to  identify  that  a  specific 
test  for  a  given  individual  has  been 
administered  and  scored) . 


Test  Data 
Test  Item  Data 
Selected  Responses 
Knowledge  Test  Score  Key 
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PROCESS : 

DATA  OUTPUT: 

STRUCTURE : 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Score  Test  and  Provide  Feedback  (continued) 


Knowledge  Test  Results  Notice 
Failed  QC  Evaluation  Notice 
Test  Score  Data 
Evaluation  Results  Data 
Pass-Fail  Log  Data 


N/A 
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PROCESS:  2.2.4,  Account  For  Off-line  Evaluation  Materials 

LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  Test  Control  Officer, 

or  other  designated  individuals,  to  control  and 
maintain  accountability  of  off-line  evaluation 
materials  (copies  of  tests,  completed  answer 
sheets,  score  keys,  etc.). 

On-line  capabilities  shall  be  provided  to 
monitor  the  receipt,  administration  and  timely 
disposition  of  offline  evaluation  materials. 

Off-line  procedures  shall  be  developed  and 
documented  which  describe  the  ways  in  which 
Test  Control  Officers  (TCOs) /other  designated 
persons  shall  manage  and  control  the  actual 
receipt,  administration  and  disposition 
of  the  off-line  evaluation  materials. 

An  on-line  log  record  shall  be  automatically 
generated  each  time  an  evaluation  event  is 
scheduled  and  the  evaluation  materials  for  the 
event  are  printed  for  off-line  administration. 

The  log  record  shall  be  used  to  monitor 
accountability  of  the  offline  materials  and  shall 
include : 

1.  Event/test  data  (Task  ID,  Objective  ID, 

Event  ID,  Test  ID,  Test  Control  Number,  type 
of  test,  date  test  was  printed,  test 
requestor's  name,  evaluator's  name  and  number 
of  test  copies  printed) , 

2.  Trainee  data  (trainee  identification  data  - 
name  and  SSAN?  and  overall  test  results 
data  -  none/passed/ failed)  for  each  trainee 
participating  in  the  evaluation  event,  and 

3.  Test  accountability  data  (suspsense  date  and 
test  disposition  status) . 
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PROCESS : 

DESCRIPTION: 


Account  for  Off-line  Evaluation  Materials 
(continued) 


(continued) 

Data  for  each  log  record  shall  be  updated  and 
maintained  in  the  following  ways: 

1.  Event/test  data  shall  be  automatically 
updated  once  the  evaluation  event  is 
scheduled  and  the  evaluation  materials 
are  printed.  Event/test  data  shall  be 
maintained  until  the  log  record  is  deleted. 

2.  Trainee  data  shall  be  automatically 
updated  when  the  event  is  scheduled  and 

the  evaluation  materials  are  printed.  Also, 
trainee  data  shall  be  automatically  updated 
once  scoring  of  the  test  for  each  trainee  has 
occurred.  Trainee  data  shall  be  maintained 
until  the  log  record  is  deleted. 

3.  Test  accountability  data  shall  be  initially 
automatically  updated;  and  shall  be  updated 
by  the  Test  Control  Officer  or  other 
designated  persons  as  necessary  to  maintain 
an  accurate  account  of  the  materials. 

The  following  data  shall  apply: 

a.  The  suspense  date  for  final  disposition 
of  materials  (automatically  established 
as  seven  days  from  date  materials  are 
printed) ,  and 

b.  The  disposition  status  of  the 
materials. 
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PROCESS : 

DESCRIPTION: 


DATA  INPUT: 


DATA  OUTPUT: 
STRUCTURE: 


Account  for  Off-line  Evaluation  Materials 
(continued) 


(continued) 

NOTE:  The  disposition  status,  for  each 
trainee,  shall  be  automatically  established 
as  "outstanding”  when  the  materials  are 
printed.  Once  each  trainee's  copy  of  the  test 
is  scored  (via  an  Optical  Mark  Reader  or 
keyboard  input) ,  the  disposition  status 
for  that  trainee  shall  be  automatically 
updated  to  "administered".  When  appropriate, 
the  Test  Control  Officer/other  designated 
persons  shall  update  the  dispositon  status 
to  "destroyed"  or  "stored". 

4 .  Once  all  copies  of  a  test  for  a  given 

evaluation  event  have  been  updated  as  being 
destroyed  or  stored,  the  log  record  has  no 
further  value  and  shall  be  automatically 
deleted. 

Log  Records  shall  be  able  to  be  printed  or  reviewed 
online,  upon  request  by  an  authorized  person  (such 
as  TCO,  evaluator,  supervisor) . 

For  details  pertaining  to  the  control,  management 
and  disposition  of  off-line  evaluation  materials, 
refer  to  the  Off-line  Evaluation  Materials 
Accountability  Procedures  documentation,  (not 
included  within  these  specifications) . 


Evaluation  Event  Data 
Log  Record  Test  Data 
Pass-Fail  Log  Data 

Evaluation  Materials  for  Accountability 
Event  Log  Record 


Event  Log  Record 
N/A 
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DATA  FLOW:  Evaluation  Event  Data 


DESCRIPTION:  Data  defining  a  scheduled  and  generated  evaluation 

event  for  an  AFS  task  or  a  Quality  Control  (QC) 
evaluation.  These  data  are  used  for  administering 
a  knowledge  test  on  line,  administering  a 
knowledge  or  performance  test  off  line  and 
accounting  for  off-line  evaluation  materials. 
Event  data  shall  enable  the  appropriate  evaluation 
materials  to  be  presented  to,  and  managed  by,  the 
appropriate  personnel. 


DATA  SOURCE:  External  Entity  1.3 


DATA  DESTINATION:  DFD  2.2,  Process  1,  2  and  4 


STRUCTURE : 


Data  elements  shall  include: 


Event  Identification  Data 


-  Event  Number  The  unique  event  number  automatically 

assigned  by  the  system  for  the  event. 

-  AOTS  Task  ID  ID  of  the  task  for  which  the  evaluation 

event  applies.  The  ID  shall  be  an 
alpha/numeric  string  of  six  characters. 

-  Task  Level  The  level  of  the  task  for  which  the 

evaluation  applies.  This  code  will 
either  be  a  T  (to  designate  task  level) 
or  the  respective  Subtask  ID  (to 
designate  the  appropriate  subtask) . 

-  Objective  ID  ID  of  the  objective  for  which  the 

evaluation  event  applies.  The  ID  is 
a  unique  numeric. 
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DATA  FLOW:  Evaluation  Event  Data  (continued) 


STRUCTURE :  ( continued ) 


Data  Items 


Descriptions 


Event  Identification  Data  (continued) 

-  Type  of  Evaluation  Designation  of  the  evaluation  event 

Event  type:  AFS  task  knowledge  or  performance 

evaluation  event  or  Q  C  evaluation 
event . 


Event  Date/Times  Date  and  time  the  event  is  scheduled  to 

begin  and  end,  if  scheduled  for  a 
specific  timeframe.  Dates  shall  be  in 
the  DDMMYYYY  format;  times  shall  be  in 
the  military  time  format  (0830,  1400, 
2300,  etc. ) . 

Evaluator's  ID  SSAN,  name  and  rank  of  the  designated 

evaluator. 


Trainee (s)  ID 


Test  ID 


Administration 

Data 


SSAN,  name  and  rank  of  each  individual 
being  evaluated. 

The  numeric  ID  of  the  test  which  is 
to  be  administered  during  the  evaluation 
event . 

The  designation  of  whether  the  test  is 
to  be  administered  on  line  or  off  line. 
(Only  applies  if  test  is  a  knowledge 
test;  a  performance  test  is  always 
administered  off  line.) 
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DATA  FLOW:  Evaluation  Materials  for  Accountability 


DESCRIPTION:  Evaluation  materials  which  were  used  to  administer 

and  score  an  off-line  knowledge  or  performance 
test  to  one  or  more  trainees  for  a  given  task 
knowledge  evaluation,  task  performance  evaluation 
or  a  Quality  Control  (QC)  evaluation  event. 

These  materials  are  received  from  the  operational 
unit  by  the  Test  Control  Officer  (TCO) ,  who  shall 
then  update  the  disposition  status  of  the 
evaluation  materials  for  each  trainee  who  was 
evaluated. 


DATA  SOURCE:  External  Entity  5 


DATA  DESTINATION:  DFD  2.2,  Process  4 


STRUCTURE:  Data  elements  shall  include: 


Test  Copies 


The  hard  copies  of  the  test  which  had 
been  printed.  (One  copy  for  each 
trainee  who  was  evaluated.) 


Knowledge  Test  The  score  key  which  was  used  to  provide 

Score  Key  feedback  to  a  trainee  who  misses  one  or 

more  test  items  on  a  knowledge  test. 


Completed  Answer  The  AOTS  answer  sheet  which  was  used  to 

Sheet (s)  administer  and/or  score  the  test  for 

each  trainee. 
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DATA  FLOW:  Evaluation  Results  Data 


DESCRIPTION:  Overall  test  results  that  are  automatically  passed 

to  processes  within  the  Management  Subsystem 
whenever  an  on-line  or  off-line  test  is  scored  for 
an  individual  (using  an  OMR  or  keyboard) .  These 
data  initiate  automatic  processes  that  update 
relevant  management  and  training  records,  and 
notify  appropriate  persons  of  the  completion  and 
overall  results  of  the  evaluation.  This  data  flow 
applies  to  the  results  of  a  Quality  Control  (QC) 
evaluation  or  an  AFS  task  evaluation. 


DATA  SOURCE: 

DATA  DESTINATION: 


DFD  2.2,  Process  3 
External  Entity  1.2 


STRUCTURE:  Data  elements  shall  include: 


Data  .  .Item 

Test  Identification 
Data 

Trainee's  ID 
Evaluator's  ID 

Overall  Test  Results 


Description 

Data  that  identifies  the  particular  test 
which  was  scored,  to  include  the 
respective  behavioral  objective  ID  and 
test  ID. 

SSAN,  name  and  rank  of  the  trainee  who 
took  the  test/performed  the  evaluation. 

SSAN,  name  and  rank  of  the  evaluator 
who  administered  the  test  (off-line  test 
only)  . 

The  "passed"  or  "failed"  results  for  the 
trainee. 
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DATA  FLOW:  Event  Log  Record 


DESCRIPTION:  An  on-line  log  record  which  is  automatically 

established  once  an  evaluation  event  has  been 
scheduled  and  the  respective  evaluation  materials 
have  been  printed  for  off-line  administration. 

The  log  record  shall  be  used  to  monitor  the 
accountability  of  the  off-line  materials.  The 
log  record  shall  be  maintained  with  other  existing 
log  records;  some  data  for  the  log  record  shall  be 
updated  automatically  by  the  system  (ie. ,  when  the 
test  is  scored  for  a  trainee) ;  other  data  shall  be 
updated  by  a  Test  Control  Officer  (TCO)  or  other 
designated  individuals.  The  log  record  shall 
exist  until  data  is  updated  reflecting  the  final  . 
disposition  of  all  appropriate  training  materials. 
A  TCO,  and  other  designated  individuals,  shall 
be  able  to  review  the  log  record  on  line,  print 
the  log  record  or  update  data  for  the  log  record. 


DATA  SOURCE: 

DATA  DESTINATION: 


DFD  2.2,  Process  4  and  Data  Store  E5 
DFD  2.2,  Data  Store  E5  and  Process  4 


STRUCTURE:  Data  elements  shall  include: 

Data  item  Description 

Event/Test  Data  Data  which  are  automatically  updated 

once  the  evaluation  event  is  scheduled 
for  which  off-line  materials  have  been 
printed.  Such  data  will  include: 

AOTS  Task  ID,  Subtask  ID  (if 
appropriate) ,  Behavioral  Objective  ID, 
Test  ID,  Test  Control  Number,  type  of 
test,  date  test  copies  were  printed, 
test  requestor's  ID,  evaluator's  ID 
and  number  of  test  copies  printed. 
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DATA  FLOW:  Event  Log  Record  (continued) 


STRUCTURE:  (continued) 

Data  Items  Descriptions 

Trainee  Data  SSAN,  name,  and  rank  of  each  airmen 

being  evaluated  and  the  overall  test 
results  (none/passed/failed)  for  each. 

Test  Accountability  Data 

Date  (DDMMYYYY)  when  the  evaluation 
materials  are  suspensed  for  final 
disposition.  This  date  shall  be 
automatically  established  as  7  days 
from  the  date  the  materials  are  printed. 

The  current  status  of  the  evaluation 
materials  for  each  trainee.  The  status 
shall  be  automatically  established  as 
"outstanding"  once  materials  are 
printed;  shall  be  automatically  updated 
to  "administered"  once  the  test  has  been 
scored;  and  shall  be  updated  by  the  TCO, 
or  other  designated  individuals,  to 
"stored"  or  "destroyed"  once  appropriate 
disposition  of  the  materials  has  been 
accomplished. 


-  Suspense  Data 


Test  Disposition 
Status 
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DATA  FLOW:  Failed  QC  Evaluation  Notice 

DESCRIPTION:  A  notice  automatically  printed  to  a  trainee's 

Unit  Commander  when  the  trainee  fails  a  Quality 
Control  (QC)  evaluation.  This  notice  advises 
that  the  trainee  was  found  to  be  unqualified  for 
a  particular  task  and  is  recommended  for  task 
decertification  and  remedial  training  before 
recertification . 


DATA  SOURCE:  DFD  2.2,  Process  3 


DATA  DESTINATION:  External  Entity  5 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 
Notice  Date 


Trainee's  ID 


Description 

DDMMYYYY  notice  was  printed  (same  day 
test  was  scored) . 

Name,  rank  and  SSAN  of  the  trainee  who 
failed  the  QC  evaluation. 


QC  Task  Data  AOTS  Task  ID  and  task  statement  which 

applies  to  the  task  for  which  the 
individual  was  QC-evaluated. 


Notice  Explanation/  The  text  which  explains  the  purpose 

Recommendation  for  the  notice. 
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DATA  FLOW:  Knowledge  Test  Results  Notice 


DESCRIPTION:  A  hardcopy  notice  which  details  an  individual's 

results  of  a  knowledge  test  administered  off  line. 
The  notice  shall  be  automatically  printed  to  the 
trainee's  supervisor,  once  the  individual's  test 
has  been  scored  (using  an  OMR  or  the  keyboard) . 


DATA  SOURCE:  DFD  2.2,  Process  3 


DATA  DESTINATION:  External  Entity  5 


STRUCTURE:  Data  elements  shall  include: 


Data  Item  Description 

Trainee's  ID  SSAN,  name,  and  rank  of  the  trainee  who 

took  the  test. 


Event/Test 
Identification  Data 


Test  score  data 


Data  which  includes  the:  evaluator's 
name.  Event  ID,  Test  Control  Number, 
event-specific  data  (Task  ID,  task 
level.  Objective  ID,  type  of  event 
and  Test  ID) ,  task/subtask  statement 
and  date  test  was  taken. 

Percent  of  correct  responses  by  test  and 
by  objective,  pass/fail  for  total  test, 
pass/ fail  by  objective,  and  test  item  ID 
for  each  incorrectly  answered  item. 
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DATA  FLOW:  Knowledge  Test  Score  Key 


DESCRIPTION:  A  printed  score  key  for  a  particular  knowledge 

test  which  was  administered  off  line.  The  key 
shall  be  used  by  a  trainee's  supervisor,  to 
provide  feedback  on  missed  test  items,  once  the 
test  is  scored.  The  key  contains  the  correct 
answer (s)  for  each  test  item  included  in  the  test. 


DATA  SOURCE: 


DFD  2.2,  Process  2 


DATA  DESTINATION:  DFD  2.2,  Process  3 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 


Description 


Test  Print  Date 


Test  Control  Number 
(TCN) 


Correct  Answers 


DDMMYYYY  the  test  copies  and  test  score 
key  are  printed. 

Number  automatically  assigned  by  the 
system  to  control  the  evaluation 
materials  which  are  printed. 

The  correct  answer (s)  for  each  test 
item  included  in  the  test. 
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DATA  FLOW:  Log  Record  Test  Data 


DESCRIPTION:  Test  identification  and  other  data  required  to 

account  for  the  distribution  and  disposition  of 
evaluation  materials  printed  for  off-line 
administration.  These  data  shall  be  automatically- 
updated,  to  create  an  on-line  event  log  record, 
once  an  offline  evaluation  has  been  scheduled  and 
the  evaluation  materials  have  been  printed. 


DATA  SOURCE: 


DFD  2.2,  Process  2 


DATA  DESTINATION:  DFD  2.2,  Process  4 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 

Test  Control  Number 
(TCN) 

Type  of  Test 
Number  of  Copies 


Test  Print  Date 


Description 

Number  automatically  assigned  by  the 
system  to  control  the  evaluation 
materials  which  were  printed.  The  TCN 
shall  not  exceed  999. 

"Performance"  or  "knowledge". 

Number  of  test  copies  printed  for  the 
event.  The  number  of  copies  should 
equal  the  number  of  trainees  who  are 
participating  in  the  evaluation  event. 

DDMMYYYY  the  evaluation  materials  were 
printed  for  offline  administration. 


210 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW: 


Pass-Fail  Log  Data 


DESCRIPTION: 


Data  which  identifies  the  overall  test  results  for 
an  individual  who  was  administered  an  off-line 
knowledge  or  performance  test.  "Passed"  or 
"failed”  shall  be  automatically  recorded  on  the 
respective  log  record  when  a  trainee's  test  is 
scored;  this  data  shall  also  cause  the  log  record 
test  status  to  be  automatically  updated  from 
"outstanding”  to  "administered”. 


DATA  SOURCE: 


DFD  2.2,  Process  3 


DATA  DESTINATION: 


DFD  2.2,  Process  4 


STRUCTURE : 


Data  elements  shall  include: 


Trainee's  ID 


Test  Control  Number 
(TCN) 


SSAN ,  name,  and  rank  of  the  trainee 
for  which  the  test  was  scored. 

Number  automatically  assigned  by  the 
system  to  control  the  evaluation 
materials  which  were  printed. 


Pass/Fail  Information  "Passed”  or  "failed”,  as  appropriate. 
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DATA  FLOW:  Selected  Responses 


DESCRIPTION:  The  trainee's  selected  responses  to  knowledge  test 

items  or  the  evaluator's  recorded  observations  for 
a  performance  test.  These  data  shall  be  recorded 
automatically  (if  the  knowledge  test  was 
administered  on  line)  or  shall  be  recorded  on  an 
answer  sheet  (if  the  knowledge/performance  test 
was  administered  off  line) .  The  selected  answers 
shall  be  used,  with  other  test/ test  item  data,  to 
score  the  test. 


DATA  SOURCE:  DFD  2.2,  Process  1  and  2 

DATA  DESTINATION:  DFD  2.2,  Process  3 


STRUCTURE : 


Data  elements  shall  include: 


Data  Item 
Examinee  ID 
Evaluator  ID 


Description 

SSAN,  name,  and  rank  of  the  examinee. 
SSAN,  name,  and  rank  of  the  evaluator. 


Examinee's  Response 
Data 


The  examinee's  response  to  each  item  on 
a  particular  knowledge  test.  (Knowledge 
test  only.) 


Evaluator's  Observations  The  evaluator's  observations  of  the 

trainee's  performance  or  product.  The 
evaluator  shall  be  required  to  record 
an  overall  ''pass"  or  ’•fail"  observation; 
the  evaluator  may  also  record 
observations  of  each  step  contained  in 
the  Performance  Evaluation  Checklist 
(PEC).  (Performance  test  only.) 
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DATA  FLOW:  Test  Data 


DESCRIPTION:  Complete  specifications  developed  or  revised  for  a 

particular  performance  or  knowledge  test. 

Test  data  result  from  processes  performed  by  the 
Evaluation  Instrument  Management  Component  (2.1); 
are  extracted  from  the  consolidated  test  data 
file  (Data  Store  E4) ;  and  are  used  for  processes 
within  the  Performance  Evaluation  Component 
(administering  a  knowledge  test  on  line, 
administering  a  knowledge/ performance  test  off 
line  and  scoring  a  test) . 


DATA  SOURCE: 

DATA  DESTINATION: 


Data  Store  E4 

DFD  2.2,  Process  1,  2,  3 


STRUCTURE:  Data  elements  shall  include: 


Data  Item 

Test  Identification 
Data 


Selected  Objectives 
and  Test  Items 


Test  Format  Data 
-  Order  of  Items 


Critical  Items 


Passriptipn 

Data  that  identifies  the  test  being 
administered  or  scored,  to  include 
the  behavioral  objective  ID  for 
which  the  test  applies  and  the  test  ID. 

Behavioral  objectives  and  test  items 
selected  for  inclusion  in  the  test. 
Behavioral  objectives  are  identified  by 
BO  IDs;  test  items  are  identified  by 
BO  ID  and  Test  Item  ID. 


Order  in  which  the  items  will  appear  in 
the  test  at  presentation  (if  the  item 
scrambling  option  in  the  test  parameters 
has  not  been  invoked) . 

Items  designated  as  critical  items. 
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DATA  FLOW:  Test  Data  (continued) 


STRUCTURE:  (continued) 

Data  Item  Description 


Test  Format  Data  (continued) 

-  Pass  Criterion  (Knowledge  test  only)  Number  of  correct 

responses  required  to  achieve  a  passing 
score  on  the  objective. 


Test  Parameter  Data 

-  Test  Use  Designation  that  the  test  is  either  the 

primary  test  of  its  type  (performance  or 
knowledge)  or  an  alternate  test. 

-  Item  Analysis  Item  analysis  capability  enabled  or 

disabled.  Default  shall  be  enabled. 


-  Maximum  Analysis 
Samples 


-  Time  Allowed 


Maximum  number  of  samples  of  test 
scores  that  will  stored  by  the  system 
for  use  in  item  analysis.  Range  shall 
be  30  to  100;  default  shall  be  50. 

(Knowledge  tests  only)  Time  allowed  the 
airman  for  completing  the  test.  Range 
shall  be  1  to  1440  minutes;  default 
shall  be  60  minutes. 


Item  Scrambling  (Knowledge  tests  only)  Enable  or 

disable  scrambling  of  items  at 
presentation.  Default  shall  be 
disabled. 


-  Test  Interruption  (Knowledge  tests  only)  Interruption 

to  be  allowed/disallowed  during  testing. 
Default  shall  be  allowed. 

-  Test  Item  Recap  (Knowledge  tests  only)  Recapitulation 

of  missed  test  items  with  correct 
response  displayed  to  be 
allowed/disallowed  during  online 
testing.  Default  shall  be  disallowed. 
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DATA  FLOW:  Test  Data  (continued) 

STRUCTURE:  (continued) 

Data  items  PsasriRtlona 

Test  Instructions  (Knowledge  test  only) .  Instructions 

which  include  the  test  purpose  and 
testing  procedures. 
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DATA  FLOW:  Test  Item  Data 


DESCRIPTION:  Data  pertaining  to  a  particular  test  item,  to 

include:  test  item  identification  data  and  test 
item  contents  (text  and  item  specifications) . 

Test  data  result  from  processes  performed  by  the 
Evaluation  Instrument  Management  Component  (2.1); 
are  extracted  from  the  Test  Item  Bank  (Data  Store 
E3) ;  and  are  used  for  processes  within  the 
Performance  Evaluation  Component  (administering  a 
knowledge  test  on  line,  administering  a 
knowledge/performance  test  off  line  and  scoring  a 
test) . 

DATA  SOURCE:  Data  Store  E3 


DATA  DESTINATION:  DFD  2.2,  Process  1,  2  and  3 


STRUCTURE:  Data  elements  include: 


Test  Item 

Identification 

Data 


Data  that  identifies  a  particular  test 
item,  including  task/ subtask  ID, 
behavioral  objective  ID  and  test  item 
ID. 


Test  Item  Type  Oral  test  guide  (OTG) ,  performance 

evaluation  checklist  (PEC) ,  or  any  of 
the  eight  types  of  knowledge  test  items: 
true/ false,  multiple  choice,  list 
multiple  choice,  matching,  constructed 
response,  limited  constructed  response, 
single  area  touch,  or  sequence  touch. 

Test  Item  Contents  Test  item  text  and  specifications.  The 

contents  depend  on  the  type  of  test 
item.  Refer  to  the  data  flows  titled: 
OTG,  PEC  or  Knowledge  Test  Item,  for 
descriptions  of  test  item  contents  data. 
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DATA  FLOW:  Test  Sample  Score  Data 


DESCRIPTION:  Data  which  result  from  scoring  a  given  sample 

of  a  knowledge  or  performance  test. 

Test  score  data  shall  be  stored  for  later 
use  by  the  Evaluation  Instrument  Management 
Component  (2.1),  for  analyzing  the  test  and' 
its  test  items  to  validate  the  materials. 

Some  test  score  data  applies  to  either  a 
performance  or  a  knowledge  test;  other  test  score 
data  depends  on  the  type  of  test  scored. 


DATA  SOURCE:  DFD  2.2,  Process  3 


DATA  DESTINATION:  DFD  2.1,  Data  Store  E6 


STRUCTURE:  Data  elements  shall  include: 

Data  Item 

Test  Identification 
Data 


Date  of  Administration 


Description 

Data  identifying  the  specific  test,  to 
include  the  task  ID,  subtask  ID  (if 
applicable) ,  behavioral  objective  ID  and 
test  ID. 

DDMMYYYY  in  which  the  test  sample  was 
scored. 


Test  Item 

Identification  Data 


Data  identifying  each  test  item 
incorporated  in  the  test,  to  include  the 
behavioral  objective  Id  and  test  item 
ID. 
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DATA  FLOW:  Test  Sample  Score  Data  (continued) 


STRUCTURE:  (continued) 


Performance  Test  Score  Data 

-  Pass/Fail  Data  The  designation  of  whether  examinee 

passed  or  failed  test. 

-  Step  Pass/Fail  Number  of  steps  failed  by  the  examinee, 

Data  specific  steps  failed,  number  of  steps 

unobserved,  specific  steps  unobserved. 

-  Critical  Step  Critical  steps  failed,  number  of  times 

each  critical  step  was  failed. 


Knowledge  Test  Score  Data 

-  Pass/Fail  Data  The  designation  of  whether  the  examinee 

passed  or  failed  the  test. 

•  Percentage  Percentage  of  items  the  examinee 

Correct  answered  correctly. 

-  Incorrect  Items  Identification  of  test  items  failed. 

-  Response  Response  alternatives  selected  for  each 

Alternative  item. 

Data 

-  Objectives  Identification  of  failed  objectives. 

Failed 

-  Critical  Items  Identification  of  failed  critical  items. 
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DATA  STORE: 

DESCRIPTION: 


DATA  SOURCE: 

STRUCTURE : 

Data  Item 

Test  Item 

Identification 

Data 
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E3 ,  Test  Item  Bank 


The  central  depository  of  all  data  for  all  test 
items  (Oral  Test  Guides,  Peformance  Evaluation 
Checklists  and  knowledge  test  items)  that  have 
been  developed  or  revised. 

As  data  for  each  test  item  are  developed, 
the  data  are  stored  within  this  central 
data  store;  as  data  are  revised,  the 
existing  data  are  extracted  from  the  file 
and  replaced  by  the  revised  data. 

Test  item  identification  data,  the  test 
item  contents  and  references  which  apply 
to  each  test  item  are  stored  within  this 
file. 

-  Once  a  test  item  is  deleted,  all  data  for 
that  test  item  shall  automatically  be 
deleted  from  this  datastore. 


Evaluation  Instrument  Management  Component  (2.1) 


An  on-line  file  which  shall  include  the  following 
data  for  each  test  item: 

Pescyiptjon 

Data  that  identifies  a  particular  test 
item,  including  the  behavioral  objective 
ID  to  which  the  test  item  applies  and 
the  test  item  ID. 
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DATA  STORE:  E3,  Test  Item  Bank  (continued) 


STRUCTURE:  (continued) 

Data  Items 
Test  Item  Type 


Test  Item  Contents 


-  OTG  contents 

—  Behavioral 
Ob j  ective 
Statement 


Descriptions 

Oral  test  guide  (OTG) ,  performance 
evaluation  checklist  (PEC) ,  or  any  of 
the  eight  types  of  knowledge  test  items: 
true/ false,  multiple  choice,  list 
multiple  choice,  matching,  constructed 
response,  limited  constructed  response, 
single  area  touch,  or  sequence  touch. 

Test  item  text  and  specifications.  The 
contents  depend  on  the  type  of  test 
item: 


The  complete  behavioral  objective 
statement  for  which  the  OTG  applies. 


Evaluator 

Info 


—  Evaluator 
Instructions 


Information  to  be  used  by  an  evaluator 
to  help  prepare  for  a  specific  task  or 
Quality  Control  evaluation.  Such 
information  will  include:  estimated 
time  to  conduct  the  evaluation; 
prerequisite  tasks  which  should  be 
trained  and  evaluated  before  the 
evaluation  is  conducted;  whether  the 
task  is  performed  by  one  individual  or 
by  a  team;  the  tools,  equipment  and 
materials  (TEMs)  required  by  the 
evaluatee;  the  TEMs  required  by  the 
evaluator;  and  any  conditions  under 
which  the  evaluation  must  be  conducted. 

Guidelines  for  administering  the 
performance  evaluation,  including  the 
wording  of  the  spoken  instructions  that 
the  evaluator  will  say  to  the  airman/ 
trainee. 
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DATA  STORE:  E3 ,  Test  Item  Bank  (continued) 


STRUCTURE:  (continued) 

Data  Items  Descriptions 

Test  Item  Contents  (continued) 

-  PEC  contents 

—  PEC  steps  The  text  for  each  PEC  step.  There  shall 

be  a  maximum  of  60  steps. 


-  Knowledge  Test  Item  contents 

—  Item  Stem  The  statement/question  setting  forth  the 

situation  to  which  an  examinee  must 
respond.  The  stem  may  contain  up  to 
three  graphics. 

Data  Type:  Alpha/numeric  string,  not  to 
exceed  3600  characters/spaces. 

—  Graphics  Data  (optional) 

Graphic  ID(s)  Alphanumeric  name(s)  or  code(s)  used  to 

call  up  the  graphic (s)  selected  by 
the  test  developer/reviser  for  use  in 
the  test  item  stem.  Up  to  three 
graphic  IDs  can  be  specified. 

Data  Type:  Alpha/numeric  string,  not 
to  exceed  10  characters/ spaces . 

Graphic (s)  The  actual  graphic (s)  selected  by  the 

developer/ reviser.  Up  to  three  graphics 
can  exist  for  the  test  item  stem. 

Presentation  Specifications  defined  by  the  test  item 

Specifications  developer/reviser  concerning  the 

placement,  scaling,  rotation  and 
coloring  of  each  selected  graphic. 

Data  Type:  X/Y  coordinates  for 
placement  data;  numeric  (1-10)  for 
scaling  factor;  numeric  (0-360  degrees) 
for  rotation  data;  and  numeric  (0-8) 
for  color  data. 
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DATA  STORE:  E3,  Test  Item  Bank  (continued) 


STRUCTURE:  (continued) 


Data  Items  Descriptions 

Knowledge  Test  Item  Contents  (continued) 


—  Response 

Alternatives 


—  Corrrect 
Answer (s) 


Feedback 


Response  alternatives  to  the  situation 
posed  in  the  item  stem  that  are  defined 
by  the  test  item  developer/reviser. 

Data  Type  for  alternatives  depends  on 
the  type  of  knowledge  test  item. 

Response  alternative (s)  designated  by 
the  test  item  developer/reviser  as  the 
correct  response  to  the  item  stem. 

Data  Type  for  correct  answer (s)  depends 
on  the  type  of  knowledge  test  item. 

Comments  by  the  test  item  developer  or 
reviser  for  each  response  alternative. 
Data  Type:  Alpha/numeric  string,  not 
to  exceed  240  characters. 


Test  Item  References  Titles,  identification  codes,  and 
and  Breakdowns  breakdowns  of  the  publications  that 

specify  performance  and  proficiency 
requirements  applicable  to  the  test 
item. 

Test  IDs  The  unique  numerical  test  IDs  for  tests 

which  incorporate  the  test  item. 
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Test 

Identification 

Data 
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E4 ,  Consolidated  Test  Data 


The  central  depository  of  all  data  for  all  tests 
that  have  been  developed  or  revised. 

As  data  for  each  test  are  developed,  the 
data  shall  be  stored  within  this  file;  as 
data  are  revised,  the  existing  data  are 
extracted  from  this  file  and  replaced  by  the 
revised  data. 

Test  identification  data,  applicable 
objectives  and  test  items,  test  format  data 
and  test  parameter  data  which  apply  to  each 
test  shall  be  stored  within  this  file. 
Additionally,  for  each  knowledge  test,  the 
on-line/off-line  instructions  shall  also  be 
included  within  this  file. 

-  Once  a  test  is  deleted,  all  data  for  that 

test  shall  automatically  be  deleted  from  this 
datastore . 


Evaluation  Instrument  Management  Component  (2.1) 


An  on-line  file  which  shall  include  the  following 
data  for  each  test: 

Data  that  identifies  a  particular  test, 
including  the  behavioral  objective  ID 
to  which  the  test  mainly  applies  and 
the  test  ID. 
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DATA  STORE:  E4 ,  Consolidated  Test  Data  (continued) 


STRUCTURE:  (continued) 


Data  Items 


Descriptions 


Selected  Objectives  Behavioral  objectives  and  test  items 

and  Test  Items  selected  for  inclusion  in  the  test. 

Each  behavioral  objective  is  identified 
by  its  respective  BO  ID;  each  test  item 
is  identified  by  its  respective  Test 
Item  ID. 


Test  Format  Data 
-  Order  of  Items 


-  Critical  Items 

-  Pass  Criterion 


Test  Parameter  Data 
-  Test  Use 


-  Item  Analysis 

-  Maximum  Analysis 
Samples 

-  Time  Allowed 


Order  in  which  the  items  will  appear  in 
the  test  at  presentation  (if  the  item 
scrambling  option  in  the  test  parameters 
has  not  been  invoked) . 

Items  designated  as  critical  items. 

(Knowledge  test  only)  Number  of  correct 
responses  required  to  achieve  a  passing 
score  on  the  objective. 


Designation  that  the  test  is  either  the 
primary  test  of  its  type  (performance  or 
knowledge)  or  an  alternate  test. 

Item  analysis  capability  enabled  or 
disabled.  Default  shall  be  enabled. 

Maximum  number  of  samples  of  test 
scores  that  will  stored  by  the  system 
for  use  in  item  analysis.  Range  shall 
be  30  to  100;  default  shall  be  50. 

(Knowledge  tests  only)  Time  allowed  the 
airman  for  completing  the  test.  Range 
shall  be  1  to  1440  minutes;  default 
shall  be  60  minutes. 
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DATA  STORE:  E4 ,  Consolidated  Test  Data  (continued) 


-  Item  Scrambling  (Knowledge  tests  only)  Enable  or 

disable  scrambling  of  items  at 
presentation.  Default  shall  be 
disabled. 


-  Test  Interruption  (Knowledge  tests  only)  Interruption 

to  be  allowed/disallowed  during  testing. 
Default  shall  be  allowed. 


-  Test  Item  Recap  (Knowledge  tests  only)  Recapitulation 

of  missed  test  items  with  correct 
response  displayed  to  be 
allowed/disallowed  during  online 
testing.  Default  shall  be  disallowed. 


Test  Instructions  (Knowledge  test  only) .  Instructions 

which  include  the  test  purpose  and 
testing  procedures . 
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DATA  STORE:  E5,  Consolidated  Log  Records 


DESCRIPTION:  A  file  containing  log  records  which  apply  to 

evaluation  events  for  which  evaluation  materials 
have  been  printed  for  off-line  administration  and 
scoring.  Each  log  record  shall  be  maintained 
until  the  Test  Control  Officer  (TCO) ,  or  another 
designated  individual,  updates  the  final 
disposition  data  for  the  evaluation  materials. 


DATA  SOURCE:  DFD  2.2,  Process  4 


STRUCTURE:  The  structure  shall  enable  on-line  maintability 

of  each  event  log  record.  This  datastore  is  an 
aggregation  of  all  existing  Event  Log  Records. 
For  each  log  record,  data  items  shall  include: 


Pats  Item 


Description 


Event/Test  Data 


Trainee  Data 


Data  which  are  automatically  updated 
once  the  evaluation  event  is  scheduled 
for  which  off-line  materials  have  been 
printed.  Such  data  will  include: 

AOTS  Task  ID,  Subtask  ID  (if 
appropriate) ,  Behavioral  Objective  ID, 
Test  ID,  Test  Control  Number,  type  of 
test,  date  test  copies  were  printed, 
test  requestor's  ID,  evaluator's  ID 
and  number  of  test  copies  printed. 

SSAN,  name,  and  rank  of  each  airmen 
being  evaluated  and  the  overall  test 
results  (none/passed/ failed)  for  each. 
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DATA  STORE:  E5,  Consolidated  Log  Records  (continued) 


STRUCTURE :  ( continued) 


Data  Items  Descriptions 

Test  Accountability  Data 


-  Suspense  Data  Date  (DDMMYYYY)  when  the  evaluation 

materials  are  suspensed  for  final 
disposition.  This  date  shall  be 
automatically  established  as  7  days 
from  the  date  the  materials  are  printed. 

-  Test  Disposition 

-  Status  The  current  status  of  the  evaluation 

materials  for  each  trainee.  The  status 
shall  be  automatically  established  as 
"outstanding”  once  materials  are 
printed;  shall  be  automatically  updated 
to  "administered"  once  the  test  has  been 
scored;  and  shall  be  updated  by  the  TCO, 
or  other  designated  individuals,  to 
"stored"  or  "destroyed"  once  appropriate 
disposition  of  the  materials  has  been 
accomplished. 
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DATA  STORE:  E6,  Consolidated  Test  Score  Data 


DESCRIPTION:  Data  stored  for  all  knowledge  and  performance 

tests  which  have  been  scored  using  an  AOTS 
keyboard  or  an  Optical  Mark  Reader. 

-  As  each  sample  of  a  test  is  scored,  data 
are  stored  within  this  central  file. 

-  Data  shall  be  aggregated  for  a  given  test,  as 
each  test  sample  is  scored,  up  to  the 
maximum  number  of  samples  designated  by  the 
test  developer. 

Data  shall  be  extracted  for  a  given  test  for 
the  purpose  of  analyzing  the  test  and  its 
test  items. 


DATA  SOURCE: 


DFD  2.2,  Process  3 


STRUCTURE : 

Data  Item 


An  on-line  file  which  shall  include  the  following 
data  for  each  test: 

Description 


Test 

Identification 

Data 


Data  that  identifies  a  particular  test, 
including  the  behavioral  objective  ID 
to  which  the  test  mainly  applies  and 
the  test  ID. 


Dates  of  Administration  Range  of  dates  during  which  the  test 

samples  were  scored. 


Number  of  Samples 


Number  of  test  samples  scored. 


Test  Item 

Identification  Data 


Data  identifying  each  test  item 
incorporated  in  the  test,  to  include  the 
behavioral  objective  Id  and  test  item 
ID. 
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DATA  STORE:  E6 ,  Consolidated  Test  Score  Data  (continued) 

STRUCTURE:  (continued) 

Data  Items  Descriptions 

Performance  Test  Score  Data 

-  Pass/Fail  Data  Total  number  of  test  failures. 

-  Step  Pass/Fail  Number  of  steps  failed  by  each  examinee, 

Data  specific  steps  failed,  number  of  steps 

unobserved,  specific  steps  unobserved. 

-  Critical  Step  Critical  steps  failed,  number  of  times 

each  critical  step  was  failed. 


Knowledge  Test  Score  Data 

-  Pass/Fail  Data 

-  Percentage 
Correct 


-  Incorrect  Items 


Total  number  of  test  failures. 

Percentage  of  correct  items  on  each 
test  administration. 

Identification  of  test  items  failed  and 
number  of  failures  on  each  item. 


-  Response 
Alternative 
Data 


Response  alternatives  selected  on  each 
item  and  number  of  times  each  response 
alternative  was  selected. 


Objectives  Identification  of  failed  objectives  and 

Failed  and  number  of  times  each  objective  was 

failed. 


Items  Identification  of  failed  critical  items 

and  number  of  times  that  each  was 
failed. 
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PROCESS:  2.3,  Apply  Training  Quality  Control 


LEVEL  OF  AUTOMATION:  Automated,  Automatic,  Manual 


DESCRIPTION:  This  process  shall  provide  for  systematic  task 

performance  evaluations  to  assess  effectiveness  of 
AOTS  training.  This  process  involves  evaluating 
individuals'  ability  to  perform  duty  position 
tasks  after  tasks  have  been  certified.  Quality 
Control  (QC)  processing  shall  enable  automatic  or 
manual  selection  and  identification  of  QC-candidate 
tasks,  evaluatees  and  evaluators;  as  well  as  the 
ability  for  a  QC  Administrator  to  initiate  a  QC 
evaluation. 

Automatic  quality  control  processing  shall  involve 
both  the  Management  Subsystem  and  the  Evaluation 
Subsystem.  The  Management  Subsystem  shall  provide 
the  Evaluation  Subsystem  with  the  task  Ids  of  tasks 
which  are  eligible  for  QC  evaluation,  as  well  as, 
task  and  individual  qualification  factors.  The 
Training  Quality  Control  Component  shall 
periodically  select  tasks  and  qualified  evaluatees 
and  evaluators  using  algorithms  whic^ support  QC 
selection  criteria.  A  pre-determined  number  of 
evaluations  shall  be  supported  for  each  periodic  QC 
selection  process.  When  possible,  these  evaluation 
selections  shall  be  spread  across  the  AFSCs 
operating  under  the  prototype  AOTS.  As  each  QC 
candidate  task-evaluatee-evaluators  combination  is 
selected,  a  notice  shall  be  generated  to  the 
appropriate  QC  Administrator.  The  QC  Administrator 
shall  use  the  notice  to  initiate  the  QC  evaluation 
event . 
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Apply  Training  Quality  Control  (continued) 


(continued) 

This  process  shall  also  support  the  manual 
selection  and  identification  of  QC  candidates. 

QC  Administrators  shall  be  allowed  to  initiate 
quality  control  evaluations  at  any  time  deemed 
appropriate,  based  on  tasks,  evaluatees  and 
evaluators  manually  identified  for  QC  processing. 

The  scheduling,  administering  and  scoring  processes 
that  apply  to  an  AFS  task  evaluation  shall 
also  apply  for  a  QC  evaluation  (i.e.,  processes 
1.3. 2. 2,  2.2.2  and  2.2.3) . 

For -each  QC  evaluation:  event  notices  shall  be 
printed  to  the  evaluatee,  evaluator ( s) ,  and 
the  evaluatee' s  supervisor  when  the  evalua¬ 
tion  event  is  scheduled;  the  performance  test 
shall  bp  administered  offline  (using  same 
evaluation  materials  used  for  evaluating  task 
performance) ;  and  test  score  results  shall  be 
passed  to  the  Management  Subsystem  to  update 
the  appropriate  training  management  records 
and  to  notify'  the  necesssary  individuals  of 
the  completion  and  overall  results  (pass/fail) 
of  the  evaluation.  In  addition,  if  a  QC 
evaluation  is  failed,  a  notice  shall  be 
automatically  generated  to  the  evaluatee 's 
Unit  Commander.  The  notice  shall  advise  the 
evaluatee  has  failed  the  QC  evaluation  and  is 
recommended  for  decertification  and  remedial 
training  before  recertification. 

The  scheduling,  administration  and  scoring 
processes  addressed  above,  as  they  pertain  to 
QC  evaluations,  are  not  further  described 
within  this  component  -  they  have  been 
described  within  the  Management  Subsystem 
(process  1.3. 2. 2)  and  the  Performance  Evalua¬ 
tion  Component  (processes  2.2.2  and  2.2.3). 
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PROCESS : 

DATA  INPUT: 

DATA  OUTPUT 

STRUCTURE : 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Apply  Training  Quality  Control  (continued) 


QC  Eligible  Tasks 

ATR  Data  for  QC  Evaluation  Selections 


QC  Notices 

QC  Event  Requirements  Data 

This  process  includes  the  following  subprocesses: 

-  Process  2.3.1,  Select  QC  Evaluation  Candidates 

-  Process  2.3.2,  Initiate  QC  Evaluation 
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Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  1  of  4 

PROCESS:  2.3.1,  Select  QC  Evaluation  Candidates 

LEVEL  OF  AUTOMATION:  Automatic,  Manual 


DESCRIPTION:  This  process  shall  enable  automatic  or  manual 

selection  and  identification  of  QC-candidate 
tasks,  evaluatees  and  evaluators. 

When  performed  manually,  a  QC  Administrator  shall 
determine  the  task,  evaluatee  and  evaluators 
(primary  and  alternate)  for  a  given  quality  control 
evaluation.  QC  candidates  may  be  selected  whenever 
considered  appropriate  by  the  QC  Administrator. 

On  a  periodic  basis,  the  automatic  QC  selection 
process  shall  randomly  identify  eligible  tasks, 
evaluatees  and  evaluators  for  QC  evaluations. 
Specific  data  shall  be  consolidated,  and  algorithms 
shall  be  invoked,  to  accommodate  Quality  Control 
(QC)  evaluation  selection  criteria. 

To  accomplish  the  automatic  QC  selection  process, 
the  Management  Subsystem  shall  identify  to  the 
Training  Quality  Control  Component  the  tasks  which 
are  eligible  for  quality  control  selection.  These 
tasks  meet  preliminary  criteria  for  quality  control 
processing. 

This  process  shall  automatically  select  QC 
evaluatee  candidates  by  examining  training  history 
data  for  each  airmen  in  each  AFS.  If  certified  on 
a  task  within  the  past  thirty  days,  an  individual 
shall  be  selected  as  a  potential  evaluatee 
candidate?  if  the  task  is  one  that  has  been 
identified  from  the  Management  Subsystem  as  being 
an  QC  eligible  task,  the  individual  shall  remain  a 
potential  evaluatee  candidate  for  that  task  and  the 
task  shall  remain  a  potential  candidate  for  QC 
evaluation. 
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PROCESS : 

DESCRIPTION: 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Select  QC  Evaluation  Candidates  (continued) 


(continued) 

This  process  shall  automatically  select  primary 
and  alternate  QC  evaluator  candidates  by  examining 
training  history  data  for  each  airmen  in  each  AFS . 
If  certified  on  a  task,  an  evaluator  shall  be 
selected  as  a  potential  candidate?  if  the  task  is 
also  an  eligible  task,  the  evaluator  shall  remain  a 
potential  candidate  as  a  QC  evaluator  for  that  task 
and  the  task  shall  remain  a  potential  candidate  for 
QC  evaluation. 

This  process  shall  then  analyze  and  match  data  for 
the  potential  tasks,  evaluatees  and  evaluators. 

The  resulting  match  shall  identify  the 
task-evaluatee-evaluator  combinations  which  are 
candidates  for  QC  evaluations.  Using  algorithms, 
the  following  selection  criteria  shall  be  invoked 
for  each  task-evaluatee-evaluator  combination: 

For  evaluatee-task  selection: 

1.  Task  must  be  on  the  Operational  Position 
Task  Requirements  (OPTR)  list  against 
which  the  airman  is  assigned, 

2.  Evaluatee  must  not  have  been  previously 
QC-evaluated  on  the  task,  and 

3.  Evaluatee  must  not  have  been  QC-evaluated 
on  any  task  within  the  past  30  days. 


F_or  evaluator-evaluatee-task  selection: 

1.  The  evaluator  must  be  assigned  to  the 
same  base/ installation  as  the  evaluatee, 

2.  The  task  must  be  one  of  the  tasks  appearing 
on  the  OPTR  list  against  which  the  evaluator 
is  assigned, 
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PROCESS: 

DESCRIPTION: 


DATA  INPUT: 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Select  QC  Evaluation  Candidates  (continued) 


(continued) 

3 .  The  evaluator  must  not  be  the  evaluatee ' s 
trainer  or  certifying  official  for  the  task, 

4.  The  evaluator  must  not  be  selected  as  the 
QC-evaluatee  for  the  task, 

5.  The  evaluator  must  not  have  been  selected  as 
an  evaluator  for  another  task  during  the 
same  time  period,  and 

6.  The  evaluator  must  be  of  equal  or  higher 
rank  than  the  evaluatee. 

Selection  criteria  shall  be  invoked,  and  the 
matching  process  shall  be  continued,  to  enable  the 
random  selection  of  QC  task-evaluatee-evaluator 
candidates  to  support  a  pre-specif ied  number  of 
QC  evaluations  (spread  across  AFSs  when  possible) . 

-  A  notice  shall  be  automatically  generated  to 
the  appropriate  QC  Administrator  when  a  valid 
task-evaluatee-evaluator  combination  is 
selected.  The  notice  shall  identify  the  task, 
evaluatee  and  the  primary/altemate  evaluators 
which  meet  the  QC  evaluation  selection 
criteria  (a  primary  evaluator  shall  be 
required;  an  alternate  evaluator  shall  be 
optional) .  The  QC  Administrator  shall  then  be 
able  to  initiate  a  QC  evaluation  based  on  the 
data  provided  by  the  notice. 

Data  pertaining  to  each  combination  shall  be 
stored  for  up  to  thirty  days,  to  preclude 
re-selection  of  the  QC  candidates  within  the 
same  thirty-day  period. 


ATR  Data  for  QC  Evaluation  Selection 
QC  Eligible  Tasks 
QC  Log  Data 
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Code  Ident.  No.  76301 
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Page  4  of  4 


PROCESS:  Select  QC  Evaluation  Candidates  (continued) 


DATA  OUTPUT:  QC  Notice 

QC  Candidate  Selection  Data 


STRUCTURE :  N/A 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  1  of  2 


PROCESS:  2.3.2,  Initiate  QC  Evaluation 


LEVEL  OF  AUTOMATION:  Automated 


DESCRIPTION:  This  process  shall  enable  a  QC  Administrator  to 

update  specific  data  to  initiate  a  Quality  Control 
(QC)  evaluation. 

The  QC  Administrator  shall  be  able  to  initiate 
a  QC  evaluation  at  any  time  deemed  appropriate  by 
providing  the  system  with  the  following  data: 

1.  The  Task  ID  which  applies  to  the  task  to  be 
evaluated, 

2 .  The  start/ stop  dates  and  times  when  the 
evaluation  event  is  to  occur, 

3.  The  SSAN  of  the  evaluatee, 

4.  The  SSAN  of  the  primary  and  alternate 
evaluator  (alternate  evaluator  shall  be 
optional) . 

This  process  shall  enable  a  QC  Administrator  to 
initiate  a  QC  evaluation  for: 

1.  Task-evaluatee-evaluator (s)  candidates 
identified  by  the  automatic  QC  eligibility 
selection  process  (2.3.1)  or 

2.  Task-evaluatee-evaluator (s)  identified  by 
some  other  means/manual  process. 

NOTE:  When  initiatiating  a  given  QC  evaluation 

which  involves  QC  candidates  identified  by 
the  automatic  QC  eligibility  selection 
process,  the  QC  Administrator  shall  be 
permitted  to  designate  different 
evaluators  than  the  evaluator (s) 
identified  on  the  QC  Notice. 
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Code  Ident.  No.  76301 
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Page  2  of  2 

PROCESS:  Initiate  QC  Evaluation  (continued) 

DESCRIPTION:  (continued) 

The  updated  QC  data  shall  then  be  passed  to  the 
Management  Subsystem,  to  accomplish  the  scheduling 
of  the  QC  evaluation  event. 

DATA  INPUT:  QC  Notice 

DATA  OUTPUT:  QC  Event  Requirement  Data 
STRUCTURE:  N/A 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


Page  1  of  2 


DATA  FLOW:  ATR  Data  for  QC  Evaluation  Selection 


DESCRIPTION:  Specific  data  automatically  examined  and  analyzed 

for  each  airman  in  an  AFS,  which  determine  whether 
the  airman  is  a  candidate  for  QC  evaluation 
selection.  The  ATR  data  shall  be  used  to  identify 
potential  QC  evaluatee  and  evaluator  candidates; 
and  to  invoke  QC  evaluation  selection  criteria 
which  enable  the  random  selection  of  QC  evaluation 
candidates . 


DATA  SOURCE: 

External  Entity  1.2 

DATA  DESTINATION: 

DFD  2.3,  Process  1 

STRUCTURE : 

Data  elements  shall  include: 

Qa.t3_ltsm 

Description 

Airman  ID 

The  airman's  full  name,  SSAN  and 

rank. 

Task  History 
Certification  Data 

The  Task  ID,  certification  date, 
and  the  SSAN  of  the  trainer,  evaluator 
and  certifying  official  for  each  task 
on  which  the  airman  is  certified. 

OPTR  ID 

The  Operational  Position  Task 

Requirement  (OPTR)  to  which  the  airman 
is  assigned  against.  The  OPTR  ID 
shall  include  the  seven  digit  numeric 
string  which  represents  the  airman's 
duty  position  number,  and  the  three 
digit  alpha/numeric  string  which 
represents  the  OPTR  version  (ie., 

STD  represents  standard  OPTR,  while 
a  3 -digit  numeral  represents  an 
individualized  version) . 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
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Page  2  of  2 


DATA  FLOW:  ATR  Data  for  QC  Evaluation  Selection  (continued) 


STRUCTURE:  (continued) 

Data  Items  Descriptions 

PAS  Code  The  PAS  code  data  for  the  airman, 

which  identifies  the  base,  MAJCOM 
and  unit  to  which  the  airman  is 
currently  assigned.  The  PAS  code 
is  an  alpha/numeric  string  of  8 
characters . 


FAC  The  Functional  Account  Code  which 

identifies  the  workcenter  to  which 
the  airman  is  currently  assigned. 

The  FAC  is  an  alpha/numeric  string 
of  4  characters. 

Training  Schedule  Data  The  data  reflected  on  the  airman's 

training  schedule,  for  each  event 
listed.  The  data  shall  include: 

-  Event  ID 

-  Airman's  participation  in  event 
(as  trainee/trainer/evaluator) 

-  Event  start/ stop  dates  and  times 

-  Training  type  identification 
(Task  ID,  level  of  task. 

Objective  ID  and  type  of  event) . 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
DatA:  31  October  1989 


DATA  FLOW:  QC  Candidate  Selection  Data 


DESCRIPTION:  Data  which  result  from  the  automatic  selection  of 

Quality  Control  (QC)  evaluation  candidates  for  a 
specific  QC  evaluation.  The  data  identify  a 
task-evalautee-evaluator  combination  which  meets  QC 
evaluation  selection  criteria.  The  data  shall  be 
maintained  with  other  task-evaluatee-evaluator 
combinations  for  30  days,  to  avoid  re-selecting  the 
QC  candidates  during  the  same  thirty-day  period. 


DATA 

SOURCE : 

DFD 

DATA 

DESTINATION: 

DFD 

2.3,  Process  l 

2.3,  Data  Store  E8 


STRUCTURE : 

Data  Item 
QC  Notice  Date 

AOTS  Task  ID 

QC  Evaluatee  ID 

QC  Evaluator  IDs 


Description 

The  date  (DDMMMYYYY)  that  QC  selection 
occurs.  The  QC  selection  date  is 
automatically  determined,  and  is  used 
to  date  the  QC  Notice  which  is  printed 
each  time  QC  selection  occurs. 

The  alpha/numeric  string  of  6  characters 
that  identifies  the  particular  task 
selected  as  a  QC  task  candidate. 

The  rank,  full  name  and  SSAN  of  the 
individual  selected  as  a  QC  evaluatee 
candidate. 

The  rank,  full  name  and  SSAN  of  the 
individuals  selected  as  QC  evaluator 
candidates.  A  primary  QC  evaluator  shall 
always  be  identified;  an  alternate  QC 
evaluator  is  optional. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  QC  Eligible  Tasks 


DESCRIPTION:  Data  applying  to  tasks  which  are  eligible  for 

Quality  Control  (QC)  evaluation  selection.  These 
tasks  are  extracted  from  the  pool  of  eligible  QC 
tasks  maintained  by  the  Management  Subsystem  and 
used  by  the  Training  Quality  Control  Component 
when  performing  the  automatic  selection  of  QC 
evaluation  candidates. 


DATA  SOURCE: 


External  Entity  1.1 


DATA  DESTINATION:  DFD  2.3,  Process  1 


STRUCTURE : 
Data  Item 
AOTS  Task  ID 

AFSC 


PAS  Code(s) 


FAC(s) 


For  each  task,  data  elements  shall  include: 
Description 

The  alpha/ numeric  string  of  6  characters 
that  uniquely  identifies  the  particular 
task. 

The  alpha/numeric  string  of  7  characters 
that  identifies  the  particular  Air  Force 
Speciality  Code  for  which  the  task 
applies. 

The  PAS  code  data  which  identifies  the 
base(s),  MAJCOM(s)  and  unit(s)  where  the 
task  is  performed.  Each  PAS  code  is  an 
alpha/numeric  string  of  8  characters. 

The  Functional  Account  Codes  which 
identify  the  workcenter (s)  where  the  task 
is  performed.  Each  FAC  is  an 
alpha/numeric  string  of  6  characters. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  QC  Event  Requirement  Data 


DESCRIPTION:  Data  which  are  updated  by  a  QC  Administrator  to 

initiate  a  Quality  Control  (QC)  event.  These  data 
are  passed  to  the  Management  Subsystem,  to  be  used 
for  scheduling  the  QC  evaluation  event. 


DATA  SOURCE: 

DATA  DESTINATION: 


DFD  2.3,  Process  2 
External  Entity  1.3 


STRUCTURE:  Data  elements  will  include: 


Data  Item 
ACTS  Task  ID 


QC  Event  Times 


Description 

An  alpha/numeric  string  of  6  characters  which 
uniquely  identifies  the  particular  task  for 
which  the  QC  evaluation  event  applies. 

The  start/ stop  dates  and  times  when  the 
evaluation  event  is  to  occur.  Dates  shall 
be  in  the  DDMMMYYYY  format,  while  times  shall 
be  in  the  military  time  format  (0830,  1400, 
2300,  etc. ) . 


Evaluatee  ID 


The  SSAN  of  the  individual  to  be  evaluated. 


Evaluator  ID(s)  The  SSAN(s)  of  the  individual (s)  who  will 

conduct  the  QC  evaluation.  A  primary 
evaluator  shall  always  be  identified;  an 
alternate  evaluator  shall  be  optional. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  FLOW:  QC  Log  Data 


DESCRIPTION:  Data  which  are  examined  each  time  the  automatic 

QC  candidate  selection  process  is  accomplished. 
The  data  are  used  to  preclude  re-selection  of. 
QC  candidate  tasks,  evaluatees  and  evaluators 
within  a  thirty-day  period. 


DATA  SOURCE: 


DFD  2.3,  Data  Store  E8 


DATA  DESTINATION:  DFD  2.3, 


Process  1 


STRUCTURE:  Data  elements  shall  include: 

Pat 4.  Item  Description 

Same  data  items/descriptions  as  for  Data  Store  E8,  QC  Log.  Refer 
to  Data  Store  E8  for  futher  details. 
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DATA  FLOW:  QC  Notice  ‘ 


DESCRIPTION:  A  printed  notice  which  shall  be  automatically 
generated  each  time  the  Quality  Control  (QC) 
evaluation  selection  process  is  accomplished  and  QC 
task-evaluatee-evaluator  candidates  have  been 
identified  for  a  specific  QC  evaluation.  The 
notice  shall  be  sent  to  the  appropriate  QC 
Administrator.  The  notice  data  may  be  used  by  the 
administrator  to  initiate  a  QC  evaluation. 


DATA  SOURCE:  DFD  2.3, 

DATA  DESTINATION:  DFD  2.3, 


Process  1 


External  Entity  5  and  Process  2 


STRUCTURE: 


Data  elements  shall  include: 


Data  Item 
QC  Notice  Date 


QC  Administrator  ID 


Description 

The  date  (DDMMMYYYY)  that  QC  selection 
occurs.  The  QC  selection  date  is 
automatically  determined,  and  is  used 
to  date  the  QC  Notice  when  the  notice  is 
printed. 

The  rank,  full  name,  SSAN  and  unit/ 
workcenter  of  the  QC  Administrator. 


QC  Evaluatee  ID  The  rank,  full  name,  and  SSAN  of  the 

individual  selected  as  a  QC  evaluatee 
candidate. 


AOTS  Task  ID  The  alpha/numeric  string  of  6  characters 

that  identifies  the  particular  task 
selected  as  a  QC  task  candidate. 
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Page  2  of  2 

DATA  FLOW:  QC  Notice  (continued) 


STRUCTURE:  (continued) 

Data  Items  Descriptions 

Task  Statement  The  statement  which  identifies  the 

behavior  and  performance  requirements 
of  the  task. 

Task  Certification  Date  The  date  (DDMMMYYYY)  the  QC  evaluatee 

candidate  was  certified  on  the  QC  task. 

QC  Evaluator  IDs  The  rank,  full  name  and  SSAN  of  the  -*» 

individual (s)  selected  as  a  QC  evaluator 
candidate.  A  primary  QC  evaluator  shall 
always  be  identified;  an  alternate  QC 
evaluator  shall  be  optional. 

QC  Notice  Purpose  The  printed  information  contained  in  the 

notice  which  identifies  the  purpose  of 
the  notice  and  the  actions  required  by 
the  recepient  (QC  Administrator) . 
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20.4.2  Training  Quality  Control  Component  Data  Stores. 
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Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  STORE:  E8 ,  QC  Log 


DESCRIPTION:  A  continuously  updated  log  of  data  which  shall  be 

maintained  for  Quality  Control  (QC)  evaluation 
selection  purposes.  The  log  shall  be  an 
accummulation  of  the  task-evaluatee-evaluator 
candidates  whic.i  result  from  the  QC  selection 
process  (accomplished  automatically 
every  seven  days) .  Data  for  selected  QC  candidates 
shall  be  maintained  on  this  log  for  a  period  of 
thirty  days. 


DATA  SOURCE:  DFD  2.3,  Process  1 


STRUCTURE : 
Data  Item 


Data  Elements  shall  include: 
Description 


QC  Notice  Dates 


The  dates  on  which  respective  tasks, 
evaluatees  and  evaluators  are  matched  and 
selected  as  QC  evaluation  candidates.  For 
each  task-evaluatee-evaluator  combination 
of  QC  candidates,  the  selection  date  is 
automatically  determined  and  used  as  the  date 
for  the  corresponding  QC  Notice.  Dates  shall 
shall  be  identified  in  the  DDMMMYYYY  format. 


AOTS  Task  IDs 


Evaiuatee  IDs 


Evaluator  IDs 


The  identification  of  tasks  selected  as 
QC  task  candidates.  Each  ID  is  an  alpha/ 
numeric  string  of  6  characters  that 
uniquely  identifies  a  particular  task. 

The  identification  of  individuals  selected 
as  QC  evaiuatee  candidates.  The  rank, 
full  name  and  SSAN  of  each  evaiuatee 
candidate  shall  be  identified. 

The  identification  of  individuals  selected 
as  QC  evaluator  candidates.  The  rank, 
full  name  and  SSAN  of  each  evaluator 
candidate  shall  be  identified. 
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System  Evaluation  Component  Processes. 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


PROCESS:  2.4,  Evaluate  System 


LEVEL  OF  AUTOMATION:  Automated,  automatic 


DESCRIPTION:  This  process  shall  enable  workcenter-level 

managers,  unit-level  training  managers,  base-level 
training  managers,  unit  commanders  and  other 
authorized  personnel  to  obtain  a  variety  of 
reports  which  can  be  used  to  monitor  training 
program  effectiveness,  training  progress  and 
training  quality. 

This  process  shall  provide  capabilities  to 
consolidate,  sort,  analyze,  and  summarize  data 
used  to  generate  standard  training  reports  as  well 
as  ad  hoc  training  reports.  This  process  shall 
also  enable  authorized  personnel  to  request 
specific  reports  suitable  for  evaluating 
individuals,  work  centers,  units  and  bases 
operating  under  the  AOTS. 


DATA  INPUT:  ATR  Data  for  Reports 

Events  Data  for  Reports 
Training  Report  Requests 


DATA  OUTPUT:  Standard/ Ad  Hoc  Training  Reports 


STRUCTURE:  This  process  includes  the  following  subprocesses: 

-  Process  2.4.1,  Consolidate/Sort  Reports  Data 

-  Process  2.4.2,  Calculate  Data  and  Generate  Reports 
Process  2.4.3,  Request  Specific  Report 
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PROCESS:  2.4.1,  Consolidate/Sort  Reports  Data 


LEVEL  OF  AUTOMATION:  Automatic 


DESCRIPTION:  This  process  shall  enable  the  system  to 

automatically  extract,  consolidate  and  sort  data 
for  reports  which  shall  be  used  to  evaluate  the 
effectiveness  of  the  AOTS. 

Data  shall  be  extracted  from  each  person's  ATR 
(including  personnel  data,  training  history  data, 
ITR  data,  position  qualification  status  data, 
etc.).  Data  shall  also  be  extracted  from  files 
which  maintain  data  for  training  and  evaluation 
events.  The  extracted  data  shall  be 
consolidated/sorted  in  a  way  to  enable  numerous 
types  of  training  reports  to  be  efficiently 
generated. 

To  provide  current  data  for  each  report,  this 
process  shall  be  accomplished  each  day/ night. 
Personnel  data  shall  be  maintained  on  a  current 
basis,  while  other  report  data  shall  be 
continually  aggregated  to  support  monthly  reporting 
periods.  Data  shall  be  sorted  and  maintained  to 
support  both  standard  training  reports  and  ad  hoc 
training  reports. 


DATA  INPUT:  ATR  Data  for  Reports 

Events  Data  for  Reports 


DATA  OUTPUT:  Sorted  Data  for  Training  Reports 


STRUCTURE :  N/A 
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PROCESS:  2.4.2,  Calculate  Data  and  Generate  Reports 


LEVEL  OF  AUTOMATION:  Automatic 


DESCRIPTION:  This  process  shall  enable  the  system  to 

automatically  calculate  data  and  generate  reports 
which  shall  identify  and  summarize  training 
effectiveness,  training  progress  and  training 
performance  under  the  AOTS. 

Standard  reports  shall  be  calculated  and 
generated  daily/nightly,  to  provide  current  and 
aggregated  data  for  evaluation.  Ad  hoc  reports 
shall  be  calculated  and  generated  on  an  "as 
requested"  basis.  Appropriate  reports  data  shall 
be  extracted,  and  algorithms  shall  be  invoked, 
to  generate  each  report. 

This  process  shall  enable  the  automatic  on-line 
storage  of  reports.  Reports  shall  be  maintained 
on  a  current  and  historical  basis. 


DATA  INPUT:  Standard  Reports  Data 

Ad  Hoc  Reports  Data 
Ad  Hoc  Report  Request  Data 


DATA  OUTPUT:  Standard  Reports 
Ad  Hoc  Report 


STRUCTURE :  N/A 
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PROCESS:  2.4.3,  Request  Specific  Report 

LEVEL  OF  AUTOMATION:  Automated 

DESCRIPTION:  This  process  shall  enable  an  authorized  person  to 

request  a  specific  training  report.  Each  requested 
report  may  be  reviewed  on  line  or  printed  for 
off-line  review. 

This  process  shall  enable: 

1.  A  given  superviser  to  request  a  training 
report  for  an  individual  whom  he/she 
supervises . 

2.  A  given  Unit  Training  Manager  to  request 
a  training  report  for  an  individual, 
workcenter  or  the  unit  for  which  he/she 
is  responsible. 

3.  A  System  Administrator  to  request  a 
training  report  for  an  individual, 
workcenter,  unit  or  base  (when  such  a 
request  is  received  from  authorized 
operational  unit  personnel,  e.g.,  Unit 
Commander,  Base  OJT) . 

This  process  shall  enable  a  report  requestor  to 
enter/select  specific  data  (ie. ,  report  name, 
printer  ID  and  print  interval)  to  obtain  a  desired 
report.  When  a  report  is  to  be  reviewed  on  line, 
the  printer  ID  and  print  interval  will  not  apply. 

This  process  shall  support  the  printing  of  a  report 
in  two  ways: 

1.  Automatic  print  upon  request, 

2.  Automatic  print  upon  specified  time  interval 
(daily,  weekly,  monthly) . 
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PROCESS:  Request  Specific  Report  (continued) 


DESCRIPTION:  (continued) 

The  following  reports,  for  an  individual,  shall  be 
made  available  for  request  from  an  authorized 
supervisor: 

1.  Position  Qualification  Status 

2 .  Upgrade  Training  Status 

3 .  CDC  status 

4 .  Evaluator  Performance  Status 

5.  Trainer  Performance  Status 

6.  Recurring  Training  Requirements 

7.  Notification  of  Impact  of  Personnel  Loss 

8 .  Ad  hoc  reports . 

The  following  reports  shall  be  made  available  for 
request  from  a  Unit  Training  Manager: 

1.  Same  individual's  reports  as  authorized  for 
a  supervisor  (1-7  above) 

2.  Workcenter/unit ' s  Position  Qualification 
Status  Summary 

3.  Workcenter/unit ' s  Upgrade  Training  Status 
Summary 

4.  Workcenter/unit' s  Upgrade  Training  Roster 

5.  Workcenter/unit ' s  Training  Event  Status 
Summary 

6.  Workcenter's  Task  Coverage  Summary 

7.  Unit's  Evaluator  Performance  Summary 

8.  Workcenter/unit ' s  Recurring  Training 
Requirements 

9 .  Ad  hoc  reports . 
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STRUCTURE: 
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Request  Specific  Report  (continued) 


(continued) 

The  following  reports  shall  be  made  available  for 
request  from  a  System  Administrator: 

1.  Any  individual's  report, 

2.  Any  workcenter/unit's  report, 

3.  Base-level  reports,  which  are  aggregations 
of  workcenter/unit  reports  data, 

4 .  Ad  hoc  reports . 

Training  Report  Request 
Standard/ Ad  Hoc  Training  Report 


Ad  Hoc  Report  Request  Data 
Standard  Report  Request  Data 

N/A 
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DATA  FLOW:  Ad  Hoc  Report 


DESCRIPTION:  A  specific  report  which  contains  requested  data 

which  has  been  calculated/analyzed  for  the  purpose 
of  evaluating  one  or  more  aspects  of  the  AOTS 
system.  The  report  contains  data  which  are  found 
on  one  or  more  standard  training  reports,  however, 
the  data  have  been  uniquely  combined  and  calculated 
or  presented  differently  than  data  which  are 
provided  by  a  standard  report. 


DATA  SOURCE :  DFD  2.4,  Process  2 


DATA  DESTINATION:  DFD  2.4,  Data  Store  E10 


STRUCTURE:  The  structure  shall  fluctuate,  depending  on  the 

data  required  by  the  specific  ad  hoc  report.  Each 
ad  hoc  report  is  unique,  therefore,  no  further 
structure  can  be  defined. 
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DATA  FLOW:  Ad  Hoc  Report  Request  Data 


DESCRIPTION:  An  automated  request  for  a  specific  ad  hoc  report. 
This  request  shall  be  updated  by  a  Systems 
Administrator  and  shall  contain  the  data  specified- 
by  the  administrator.  The  updated  request  shall  be 
calculated  by  the  system  during  the  daily/nightly 
reports  generation  process,  and  the  respective 
ad  hoc  report  shall  be  generated. 


DATA  SOURCE: 

DATA  DESTINATION: 


DFD  2.4,  Process  3 
DFD  2.4,  Process  2 


STRUCTURE:  The  structure  shall  fluctuate,  depending  on  the 

data  required  by  the  specific  ad  hoc  report.  Each 
ad  hoc  report  is  unique,  therefore,  no  further 
structure  can  be  defined. 
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DATA  FLOW:  Ad  Hoc  Reports  Data 


DESCRIPTION:  Data  used  to  calculate  and  generate  an  ad  hoc 

training  report.  These  data  are  extracted  from  the 
data  being  maintained  for  all  reports. 


DATA 

SOURCE: 

DFD 

2.4, 

Data  Store  E9 

DATA 

DESTINATION: 

DFD 

2.4, 

Process  2 

The  structure  shall  fluctuate,  depending  on  the 
data  required  by  the  specific  ad  hoc  report.  Each 
ad  hoc  report  is  unique,  therefore,  no  further 
structure  can  be  defined. 


STRUCTURE : 
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DATA  FLOW:  ATR  Data  for  Reports 


DESCRIPTION:  Data  automatically  extracted,  on  a  daily/nightly 

basis,  from  each  person's  Airman  Training  Record 
(ATR)  which  shall  be  consolidated,  sorted  and 
analyzed  for  the  purpose  of  providing  a  variety 
of  standard  and  ad  hoc  training  reports. 


DATA  SOURCE:  DFD  2.4,  External  Entity  1.2 


DATA  DESTINATION:  DFD  2.4,  Process  1 


STRUCTURE:  The  structure  of  this  data  flow  shall  enable  the 

collection  of  all  ATR  data  for  each  person 
operating  under  the  AOTS.  Data  elements  shall 
be  applicable  for  each  person  and  include: 


Data  itep 

Personnel  Data 
for  Reports 


Training  History 
Data  for  Reports 


Description 

Data  extracted  from  an  ATR  which 
identifies  the  name  and  other  personnel 
information  pertaining  to  the  individual. 
Such  data  will  include:  full  name,  rank, 
SSAN,  PAFSC,  DAFSC,  CAFSC,  2AFSC,  3AFSC, 
4AFSC,  position  number,  duty  title,  unit, 
base,  PAS  code,  FAC  and  AF  component. 

Data  extracted  from  an  ATR  which 
identifies  the  tasks  and  other  training 
requirements  for  which  the  individual  is 
qualified.  General  history  data  are  also 
extracted.  Such  data  will  include: 


General  history  data:  Training  Status 
Code,  date  entered/completed/withdrawn 
training  and  date  initially  entered 
re-training 
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DATA  FLOW:  ATR  Data  for  Reports  (continued) 


STRUCTURE:  (continued) 

Data  Items  Descriptions 

Training  History  Data  for  Reports  (continued) 

-  Task  history  data  (for  each  task) : 

Task  ID,  Task  Version,  date  individual 
was  certified,  task  statement,  trainer's 
SSAN/rank/full  name,  evaluator's 
SSAN/rank/full  name,  certifying 
official's  SSAN/rank/full  name,  and 
installation  where  individual  was 
certified  on  task 

-  Other  training  history  data  (for  each 
ancillary  course,  additional  duty 
course,  contingency  task,  ECI/CDC 
course,  PME  course,  and  formal  training 
course) :  Course  ID  (or  task  ID  or 
course  title,  as  apppropriate) ,  date 
individual  completed  the  training, 

and  installation  where  individual 
completed  the  training 

Position  Data  extracted  from  an  individual's  ATR 

Qualification  which  identifies  the  status  of  each  task 

Data  for  Reports  required  by  the  Operational  Position  Task 

Requirements  (OPTR)  against  which  the 
individual  is  assigned.  Data  for 
each  task  includes:  Task  ID,  Task  Version 
number,  da-e  certified  (if  applicable) , 
date  recertification  is  due  (if  applicable), 
current  status  of  task  (none,  in  progress, 
certified,  etc.),  and  task  statement. 
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DATA  FLOW:  ATR  Data  for  Reports  (continued) 


STRUCTURE : 
Data  Items 


(continued) 

Descriptions 


Individual 
Training 
Requirements 
Data  for  Reports 


Data  extracted  from  an  individual's  ATR 
which  identifies  the  training  requirements 
pertaining  to  the  individual.  General  ITR 
data,  task  training  requirements  and  other 
training  requirements  shall  be  extracted, 
and  include: 

-  General  ITR  data:  OPTR  ID,  primary 
trainer's  SSAN/rank/full  name,  and 
primary  evaluator's  SSAN/rank/full  name 

-  Task  requirements  data  (for  each  task) : 
Task  ID,  task  statement  and  current 
status  of  the  task  (none,  in  progress, 
awaiting  certification,  etc.) 

-  Other  training  requirments  data  (for 

each  ancillary  course,  additional  duty 
course,  contingency  task,  ECI/CDC 
course,  PME  course  and  formal  training 
course) :  Course  ID  (or  course  title 

or  Task  ID,  as  appropriate)  and  current 
status  of  the  requirement  (none  or  in 
progress) 
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DATA  FLOW:  Events  Data  for  Reports 


DESCRIPTION:  Data  which  identifies  training  and  evaluation 

events  which  have  occurred,  are  in  progress  or  have 
been  completed  during  a  specified  timeframe. 

Such  data  shall  be  consolidated,  sorted  and 
analyzed,  on  a  daily/nightly  basis,  for  the  purpose 
of  providing  a  variety  of  standard  and  ad  hoc 
training  reports.  Events  data  shall  also  be 
aggregated  to  support  monthly  reporting  periods. 


DATA 

SOURCE : 

DFD  2.4, 

External  Entity  1.3 

DATA 

DESTINATION: 

DFD  2.4, 

Process  1 

STRUCTURE:  The  structure  of  this  data  flow  shall  enable  the 

collection  of  all  data  for  each  event.  Data 
elements  are  applicable  for  each  event  and  include: 


Data  Item 
Event  ID 

Event  Status 

Event  Times 
Training  Type 

Event  Type 


Description 

A  numeric  string,  up  to  four  digits, 
identifying  a  particular  training/ evaluation 
event . 

The  initial/current  status  for  the  event  (i.e., 
scheduled,  assigned,  reschedule,  reassigned, 
none,  and  complete)  within  the  reporting 
period. 

Start  dates/times  for  the  initial  and  current 
status  for  the  event. 

Type  of  training  (ie.,  AFS  task  training, 
ancillary,  additional  duty,  or  contingency 
task) 

Type  of  event  (ie.,  knowledge  training  or 
performance  training) .  This  information 
only  applies  to  AFS  task  training  events. 
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DATA  FLOW:  Events  Data  for  Reports  (continued) 


STRUCTURE.:  (continued) 

Data  Items  Description 

Course/Task  ID  The  Course  ID  or  Task  ID  which  identifies 

the  particular  course  or  task  which  applies  to 
the  event. 


Task  Level 


Trainee's  ID 


Trainer's  or 
Evaluator's  ID 


Data  identifying  the  task  breakdown  level 
for  which  the  event  applies  (i.e.,  task-level 
or  subtask-level) .  This  information  only 
applies  to  AFS  task  training  events. 

SSAN  of  each  trainee  for  which  the  event 
applies. 

SSAN  of  the  designated  trainer  or  evaluator  for 
the  event. 


Overall  Results  Data  identifying  whether  or  not  an  event  was 

passed  or  failed.  This  information  only 
applies  to  AFS  task  training  events  which  have 
been  completed.  For  evaluation  events,  the 
data  shall  also  include  the  test  score  for  each 
trainee,  when  such  data  are  available. 


Cancellation  Data  identifying  the  reason  for  cancelling  an 

Reason  event . 
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DATA  FLOW:  Sorted  Data  for  Training  Reports 


DESCRIPTION:  Data  which  has  been  consolidated  and  sorted  for  the 

purpose  of  statistical  analysis  and  generation  of 
training  reports.  These  data  shall  be 
automatically  sorted  on  a  daily/nightly  basis  and 
shall  be  added  to  other  reports  data  previously 
consolidated/sorted  for  a  given  reporting  period. 


DATA  SOURCE:  DFD  2.4,  Process  1 


DATA  DESTINATION:  DFD  2.4,  Data  Store  E9 


STRUCTURE : 


The  structure  of  this  data  flow  shall  enable  the 
sorted  data  to  be  aligned  with  other  data  which 
is  maintained  for  the  purpose  of  training  reports. 
The  data  is  an  aggration  of  data  from  two  other 
data  flows:  ATR  Data  for  Reports  and  Events  Data 
for  Reports. 


Same  items/descriptions  as  for  ATR  Data  for  Reports  and  Events 
Data  for  Reports.  Refer  to  these  data  flows  for  further  details. 
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DATA  FLOW:  Standard  Report  Request  Data 


DESCRIPTION:  An  automated  request  to  access  a  specific  standard 

training  report.  This  request  may  be  submitted 
at  any  time,  by  any  authorized  person.  The  request 
includes  the  selection  of  the  specific  report  and 
other  data  which  identifies  how  the  report  is  to 
be  reviewed  (display  on  line,  or  print  off  line) . 


DATA  SOURCE: 


DFD  2.4,  Process  3 


DATA  DESTINATION:  DFD  2.4,  Data  Store  E10 


STRUCTURE:  Data  elements  shall  include: 

Data,  Item  Description 

Title  of  Standard  The  name  identifying  the  specific  training 

Report  report  desired.  Titles  for  standard  reports 

shall  include:  Position  Qualification 
Status  and  Summary;  Workcenter  Task 
Coverage;  Notification  of  Impact  of 
Personnel  Loss;  Upgrade  Training  Status, 
Roster  and  Summary;  Evaluator  Performance 
Status  and  Summary;  Training  Event  Status 
Summary;  Trainer  Performance;  Recurring 
Training  Requirements  and  Summary;  and 
CDC  Status.  This  data  item  shall  be 
specified  by  an  individual’s  selection  from 
a  menu  or  prompt. 
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DATA  FLOW:  Standard  Report  Request  Data  (continued) 


STRUCTURE:  (continued) 


Data  Items 


Descriptions 


Reporting  Level 
Data 


The  organizational  level/ identification  for 
which  the  report  applies.  There  shall  be 
four  reporting  levels:  Individual, 
Workcenter,  Unit  and  Base.  This  data  item 
shall  be  specified  by  an  individual's 
selectin  from  a  menu  or  prompt. 


For  an  Individual  Standard  Report,  the 
SSAN  of  the  individual  shall  be  also 
specified 


-  For  a  Workcenter  Standard  Report,  the 
PAS  Code  and  FAC  for  the  workcenter 
shall  be  also  specified 

-  For  a  Unit  or  Base  Standard  Report,  the 
PAS  shall  be  also  specified 

Report  Print  The  report  shall  be  automatically  printed 

Interval  based  on  the  interval  chosen  by  the 

requestor.  The  interval  shall  be  one  of  the 
following:  upon  request,  daily,  weekly  or 
monthly. 


Print  Location  The  Printer  ID  where  the  report  is  to  be 

printed.  The  Printer  ID  is  an  alpha/ numeric 
item,  not  to  exceed  seven  characters. 
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DATA  FLOW:  Standard  Reports 

DESCRIPTION:  The  various  AOTS  standard  training  reports  which 
are  generated  on  a  daily/nightly  basis. 

DATA  SOURCE:  DFD  2.4,  Process  2 

DATA  DESTINATION:  DFD  2.4,  Data  Store  E10 


STRUCTURE:  The  structure  of  this  data  flow  shall  support  each 

standard  training  report  which  is  generated.  For 
personnel/organizations  operating  under  the  AOTS, 
there  shall  be  (at  a  minimum) :  seven  standard 
reports  which  apply  to  each  individual;  six 
standard  reports  which  apply  to  each  workcenter; 
six  standard  reports  which  apply  to  each  unit;  and 
six  standard  reports  which  apply  to  each  base. 


Same  data  items/descriptions  as  for  Data  Store  E10,  Training 
Reports.  For  further  details,  refer  to  the  data  items  which 
apply  to  each  standard  report  described  within  Data  Store  E10. 
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DATA  FLOW:  Standard  Reports  Data 


DESCRIPTION:  Data  used  when  calculating  and  generating  various 

AOTS  standard  training  reports.  These  data  are 
extracted  from  the  data  being  maintained  for  all 
reports. 


DATA  SOURCE :  DFD  2.4,  Data  Store  E9 


DATA  DESTINATION:  DFD  2.4,  Process  2 


STRUCTUE:  The  structure  of  this  data  flow  shall  provide 

the  appropriate  data  used  to  generate  each  standard 
training  report.  For  personnel/organizations 
operating  under  the  AOTS,  there  shall  be  (at  a 
minimum) :  seven  standard  reports  which  apply  to 
each  individual;  six  standard  reports  which  apply 
to  each  workcenter;  six  standard  reports  which 
apply  to  each  unit;  and  six  standard  reports  which 
apply  to  each  base. 


Pat a  Ifcems  Description 

Same  data  items/ descript ions  as  for  ATR  Data  for  Reports  and 
Events  Data  for  Reports.  Refer  to  these  data  flows  for  further 
details. 
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DATA  FLOW:  Standard/ Ad  Hoc  Training  Report 


DESCRIPTION:  The  specific  standard  training  report  or  ad  hoc 

training  report  which  was  requested  by  an 
authorized  person.  The  report  may  be  reviewed 
on  line  by  the  report  requestor,  or  may  be  printed 
at  a  specified  printer. 


DATA  SOURCE: 


DFD  2.4,  Data  Store  E10 


DATA  DESTINATION:  DFD  2.4,  Process  3  and  External  Entity  5 


STRUCTURE:  The  structure  of  this  data  flow  shall  support  any 

requested  standard  report,  as  well  as  any  requested 
ad  hoc  report.  Data  elements  will  include: 

Data  Items  Description 

For  a  specific  standard  training  report,  refer  to  the  data  items 
and  descriptioi.s  provided  for  Data  Store  E10,  Training  Reports. 
The  contents  of  each  report:  has  been  described  within  Data  Store 
E10. 

For  an  ad  hoc  report,  the  data  items/ descriptions  are  unique.  No 
further  details  can  be  provided  since  each  report's  contents 
shall  be  specific. 
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DATA  FLOW:  Training  Report  Request 


DESCRIPTION:  A  verbal  or  written  request  for  a  specific  training 
report.  The  request  is  received  from  an 
authorized  person  within  an  operational  unit,  such 
as  Base  OJT  or  unit  commander.  The  request  is 
received  by  the  Systems  Administrator,  who  then 
updates  the  request  for  the  report  into  the  system. 


DATA  SOURCE: 


External  Entity  5 


DATA  DESTINATION:  DFD  2.4,  Process  3 


STRUCTURE:  The  structure  shall  support  a  request  for  a 

standard  report  or  an  ad  hoc  report.  Data  elements 
will  include: 


DESCRIPTIONS 


For  a  standard  training  report: 


Title  of  Standard  The  name  identifying  the  specific  training 
Report  report  desired. 


-  Reporting  Level  The  organizational  level/ identification  for 
Data  which  the  report  applies.  There  shall  be 

four  reporting  levels:  Individual, 
Workcenter,  Unit  and  Base. 


-  For  an  individual  standard  report,  the 
name  or  SSAN  of  the  individual  must  be 
provided 


For  a  workcenter,  unit  or  base  standard 
report:  the  name  of  the  organization 
must  be  provided;  or  the  PAS  Code  (for 
workcenter,  unit  and/or  base)  and  the 
FAC  (for  workcenter)  must  be  provided 
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DATA  FLOW:  Training  Report  Request  (continued) 


STRUCTURE:  (continued) 


Data  Items 


Descriptions 


Report  Print 
Interval 


The  report  can  be  printed  on  a  one-time 
basis  (upon  request)  or  a  repeated  basis 
(daily,  weekly  or  monthly) . 


-  Print  Location  The  Printer  ID  where  the  report  is  to  be 

printed.  A  workcenter's  printer  ID  may  be 
specified,  or  the  report  shall  be  printed 
directly  to  the  Systems  Administator  (who 
shall  then  forward  the  report  to  the 
requesting  individual) . 


For  an  ad  hoc  training  report: 

The  data  items/descriptions  shall  be  unique  for  each  ad  hoc 
report  request,  therefore,  no  further  details  can  be  provided. 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


DATA  STORE: 

DESCRIPTION: 

DATA  SOURCE: 

STRUCTURE : 

Data  Items 


Specification  Number  70S647300B 
Code  Ident.  No.  76301 
Date:  31  October  1989 


E9 ,  Consolidated  Training  Reports  Data 


Data  maintained  for  the  purpose  of  generating 
training  reports  to  evaluate  the  AOTS.  These 
data  have  been  consolidated  and  sorted,  to  enable 
extraction  and  statisical  analysis  to  occur  as 
required  for  each  training  report.  Data  shall 
be  maintained  on  a  current  basis,  and  shall 
support  monthly  reporting  periods. 


DFD  2.4,  Process  1 


The  structure  of  data  store  shall  be  one  or  more 
files  which  contain  tables  for  sorting  and  listing 
the  various  data  elements  required  for  all  reports. 

Description 


Same  items/ descriptions  as  for  ATR  Data  for  Reports  and  Events 
Data  for  Reports.  Refer  to  these  data  flows  for  further  details. 
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DATA  STORE:  E10,  Training  Reports 


DESCRIPTION:  A  file  which  contains  current  and  historical 

training  reports  which  have  been  generated  for  the 
purpose  of  evaluating  training  performance, 
progress,  ^lality  and  effectiveness  for  each 
person,  workcenter,  unit  and  base  operating  under 
the  AOTS . 


DATA  SOURCE:  DFD  2.4,  Process  2 


STRUCTURE:  The  structure  of  this  file  shall  enable  different 

types  of  reports  (both  standard  and  ad_hoc)  to  be 
maintained.  Data  elements  shall  be  specific  for 
each  report  and  shall  include: 


FOR  INDIVIDUAL  POSITION  QUALIFICATION  STATUS  REPORT: 


Report  Title  Name  of  report. 

Report  Date  Date  (DDMMMYYYY)  that  report  was  generated. 

Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 

Personnel  Data  Data  extracted  from  the  personnel  data  being 

for  Report  maintained  in  an  individual's  ATR.  Such 

data  will  include:  Component,  rank,  name, 
SSAN ,  base,  unit,  workcenter,  CAFSC,  OPTR  ID 
and  OPTR/Duty  Title. 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

INDIVIDUAL  POSITION  QUALIFICATION  STATUS  REPORT  (continued) : 

Position  Data  extracted  from  the  individual's  ATR,  and 

Qualification  calculated,  which  identifies  the  individual's 

Status  Data  for  status  and  progress  towards  position 
Report  qualification.  Such  data  will  include: 

-  tasks  required  for  position  qualification 
(number  required;  number  qualified; 
percent  qualified) 

-  time  in  training  for  position 
qualification  (actual  time  of  individual; 
cumulative  average  for  all  trainees) 

-  testing/ evaluations  accomplished  for 
position  qualification  (knowledge  tests 
completed,  passed  and  percentage  passed; 
performance  evaluations  completed,  passed 
and  percentage  passed;  QC  evaluations 
completed,  passed,  percentage  passed, 
failed  resulting  in  task  decertification 
and  percentage  decertified) 


FOR  INDIVIDUAL  UPGRADE  TRAINING  STATUS  REPORT: 


Report  Title 
Report  Date 

Personnel  Data 
For  Report 


Name  of  report. 

The  date  (DDMMMYYYY)  the  report  was 
generated. 

Data  extracted  from  the  personnel  data  being 
maintained  in  an  individual's  ATR.  Such 
data  will  include:  Component,  rank,  name, 
SSAN,  base,  unit,  workcenter,  CAFSC,  DAFSC, 
PAFSC,  2AFSC,  TSC,  OPTR  ID,  OPTR/Duty  Title, 
date  entered/completed/withdrawn  training 
and  date  initially  entered  retraining. 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

INDIVIDUAL  UPGRADE  TRAINING  STATUS  REPORT  (continued) 


PME  Data  for 
Report 


Position 
Qualification 
Task  Data  for 
Report 


Data  extracted  from  the  individual's  ATR 
which  identifies  the  PME  courses  the 
individual  has  completed  or  is  required  to 
complete.  For  each  PME  course,  the  course 
title  and  completion  date  (if  applicable) 
are  reported. 

Data  extracted  from  the  individual's  ATR, 
and  calculated,  which  identifies  the 
individual ' s  upgrade  training  status  in 
regards  to  task  training.  This  data  will 
include : 

-  Number  of  tasks  required  for  position 
qualification, 

-  Number  of  tasks  for  which  training  has 
been  completed  for  individual,  and 

-  Percentage  of  tasks  completed. 


FOR  INDIVIDUAL  CDC  STATUS  REPORT: 


Report  Title 
Report  Date 


Personnel  Data 
For  Report 


Name  of  report. 

The  date  (DDMMMYYYY)  the  report  was 
generated. 

Data  extracted  from  the  personnel  data  being 
maintained  in  an  individual's  ATR.  Such 
data  will  include:  Component,  rank,  name, 
SSAN,  base,  unit,  workcenter,  CAFSC,  DAFSC , 
PAFSC,  2AFSC,  TSC,  OPTR  ID,  OPTR/Duty  Title, 
and  date  entered/completed/withdrawn 
training. 
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DATA  STORE:  ’  Training  Reports  (continued) 


STRUCUTURE:  '(continued) 

Data  Items  Descriptions 

INDIVIDUAL  CDC  STATUS  REPORT  (continued) 

CDC  Status  Data  Data  extracted  from  the  individual's  ITR 
For  Report  which  identifies  the  individual's  current  CDC 

status  and  estimated/ actual  CDC  completion. 
Such  data  will  include:  Course  number,  date 
course  was  requested,  date  course  was 
received,  date  individual  was  enrolled  in 
course,  date  of  estimated  completion,  volume 
review  exercise  data  (volume,  date  started, 
estimated/actual  completion  date  and  score) , 
and  course  examination  data  (test  date  and 
test  score) . 


FOR  INDIVIDUAL  EVALUATOR  PERFORMANCE  REPORT: 

Report:  Title  Name  of  report. 

Report  Date  Date  (DDMMMYYYY)  that  report  was  generated. 

Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 

Personnel  Data  Data  extracted  from  the  personnel  data  being 

For  Report  maintained  in  an  individual's  ATR.  Such  data 

will  include:  Component,  rank,  name,  SSAN, 
base,  unit,  workcenter,  CAFSC,  OPTR  ID  and 
OPTR/Duty  Title. 
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DATA  STORE:  Training  Raports  (continued) 


STRUCTJTURE:  (continued) 

Pa  t  a,  ifcaaa  BaagcigLigna 

INDIVIDUAL  EVALUATOR  PERFORMANCE  REPORT  (continued) 


Evaluator  Data  extracted  froa  the  individual's  ATR 

Performance  (training  history)  and  froa  other  database 

Data  for  Report  files,  which  identifies  the  individual's 

ability  to  evaluate  trainees'  performance  of 
tas)cs.  The  extracted  data  shall  also  be 
calculated  to  generate  the  report  and  will 
include: 


-  number  of  tasks  the  individual  is 

'  qualified  to  evaluate  (individual  is 
qualified  to  evaluate  any  task  for  which 
he/sh*  is  certified) , 

-  number  of  performance  evaluations  the 
individual  has  conducted, 

-  number  of  performance  evaluations  which 
the  individual  conducted  and  which  were 
failed  by  the  evaluate* ( s) ,  and 

-  percentage  of  evaluations  that  were 
failed. 


FOR  INDIVIDUAL  TRAINER  PERFORMANCE  REPORT: 

Report  Title  Name  of  repc-*:. 

Report  Date  Date  (DDMMMYY*.,  that  report  was  generated. 

Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

INDIVIDUAL  TRAINER  PERFORMANCE  REPORT  (continued) 


Personnel  Data 
For  Report 


Trainer 
Performance 
Data  for  Report 


Data  extracted  from  the  personnel  data  being 
maintained  in  an  individual's  ATR.  Such  data 
will  include:  Component,  rank,  name,  SSAN, 
base,  unit,  workcenter,  CAFSC,  OPTR  ID  and 
OPTR/Duty  Title. 

Data  extracted  from  the  individual's  ATR 
and  from  other  database  files,  and  then 
calculated,  which  identifies  the  individual' 
ability  to  train  AFS  tasks.  Such  data  will 
include: 


-  number  of  tasks  the  individual  has 
trained  withirf  the  reporting  period 

-  number  of  trainees  the  individual  has 
trained  during  the  reporting  period 

-  data  for  each  task  the  individual  has 
trained 


—  Number  of  knowledge  tests 

administered,  number  failed  and 
percentage  failed 

—  Number  of  performance  tests 

administered,  number  failed  and 
percentage  failed 

—  Number  of  Quality  Control  (QC) 
evaluations  administered,  number 
failed  and  percentage  failed 

-  Number  of  tasks  in  which  training 
resulted  in  task  qualification 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

FOR  INDIVIDUAL  RECURRING  TRAINING  REQUIREMENTS  REPORT: 


Report  Title  Name  of  report. 

Report  Date  Date  (DDMMMYYYY)  that  report  was  generated. 


Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 


Personnel  Data 
For  Report 


Recurring 
Training 
Requirements 
for  Report 


Data  extracted  from  the  personnel  data  being 
maintained  in  an  individual's  ATR.  Such  data 
will  include:  Component,  rank,  name,  SSAN, 
base,  unit,  workcenter,  CAFSC,  OPTR  ID  and 
OPTR/ Duty. Title. 

Data  extracted  from  the  person's  ITR 
identifying  the  recurring  tasks  and  Other 
Training  Requirements  (OTRs)  which  the 
individual  must  accomplish.  Each  task 
is  identified  by  its  Task  ID  and  each 
OTR  is  identified  by  its  Course  ID,  course 
name  or  Task  ID  (contingency  tasks) .  The 
status  of  each  requirement  (none,  scheduled, 
in  progress,  awaiting  refresher  training, 
etc)  will  also  be  identified. 
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DATA  STORE:  Training  Reports  (continued) 

STRUCUTURE:  (continued) 


Data  Items  Descriptions 

FOR  INDIVIDUAL  NOTIFICATION  OF  IMPACT  OF  PERSONNEL  LOSS  REPORT: 


Report  Title 

Report  Date 

Personnel  Data 
For  Report 


Impact  of  Loss 
Data  for  Report 


Name  of  report. 

The  date  ( DDMMMYYYY)  report  was  generated. 

Data  extracted  from  the  personnel  data  being 
maintained  in  an  individual's  ATR.  Such  data 
will  include:  Component,  rank,  name,  SSAN, 
base,  unit,  workcenter,  CAFSC,  OPTR  ID  and 
OPTR/Duty  Title. 

Data  extracted  from  the  individual’s  ATR 
which  identifies  the  tasks  for  which  the 
individual  is  qualified  (certified) .  Other 
data  are  also  extracted  from  each  person's 
ATR  within  the  workcenter,  and  calculated, 
to  generate  the  report.  Data  reported  will 
include: 

-  Number  of  persons  assigned  within  the 
workcenter  wno  are  operating  under  the 
AOTS 

-  List  of  tasks  which  the  losing  individual 
is  qualified  to  perform.  For  each  task 
listed,  the  following  data  will  be 
reported: 

—  Task  ID 
—  Task  Statement 

—  Number  of  workcenter  personnel  who 
are  qualified  on  the  task 
—  Percent  of  workcenter  personnel  who 
are  qualified  on  the  task 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Pwcriptiona 

FOR  WORKCENTER  POSITION  QUALIFICATION  STATUS  SUMMARY  REPORT: 


Report  Title 


Name  of  report. 


Report  Date 


The  date  ( DDMMMYYYY)  report  was  generated. 


Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 


Workcenter  Data  The  data  identifying  the  workcenter  for 
for  Report  which  the  report  is  generated.  Such  data 

will  include:  Component,  base,  unit, 
workcenter  and  AFS. 


Workcenter  Posn 
Qualification 
Statue  Data  for 
Report 


Data  which  are  aggregrated  from  various 
ATRs  of  persons  assigned  to  the  workcenter, 
and  then  calculated,  to  generate  this  report 
Such  data  will  include: 

-  Number  of  personnel  assigned,  number 

of  personnel  who  are  position  qualified 
and  percent  of  persons  who  are  position 
qualified 

-  Time  in  training  for  position 
qualification 

—  cumulative  average  for  all  trainees 

—  number/percent  qualifying  in  less 
than  average  time  during  reporting 
period,  and  number/percent 
qualifying  in  greater  than  average 
time  during  reporting  period 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

WORKCENTER  POSITION  QUALIFICATION  STATUS  SUMMARY  REPORT  (con't) 

-  Task  knowledge  testing  data  for  the 
reporting  period  (number  of  knowledge 
tests  administered,  number  passed  and 
percentage  passed) 

-  Task  performance  evaluations  data  for 
the  reporting  period  (number  of 
evaluations  administered,  number  passed 
and  percentage  passed) 

-  Quality  Control  (QC)  evaluations  data 
for  the  reporting  period  (number  of 
QC  evaluations  administered,  number 
passed,  percentage  passed,  number 
failed  resulting  in  task  decertifica¬ 
tion  and  percentage  decertified) 


FOR  WORKCENTER  UPGRADE  TRAINING  STATUS  SUMMARY  REPORT: 


Report  Title  Name  of  report. 

Report  Date  The  date  (DDMMMYYYY)  report  was  generated. 

Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 


Workcenter  Data 
for  Report 


The  data  identifying  the  workcenter  for 
which  the  report  is  generated.  Such  data 
will  include:  Component,  base,  unit, 
workcenter  and  AFS. 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

WORKCENTER  UPGRADE  TRAINING  STATUS  SUMMARY  REPORT  ( continued; 

Upgrade  Training  Data  which  are  aggregated  froa  various 
Data  for  Report  ATRs  of  persons  assigned  to  the  wcrkcenter, 

and  then  calculated,  to  generate  this  report. 
Such  data  will  include: 

-  Number  of  personnel  assigned:  in 
upgrade  training,  in  3  level  training, 

in  5  level  training,  in  7  level  training, 
in  3  level  retraining,  in  5  levol 
retraining,  in  7  level  retraining;  and 
percentage  of  assigned  personnel  in 
upgrade  training 

-  CDC  enrollment  data:  number  of  persons 
enrolled  in  CDCs,  number  of  persons 
completing  CDCs  within  reporting  period, 
number  of  persons  failing  CDC  course 
examinations  within  reporting  period  and 
percentage  of  persons  completing  CDCs 
during  the  reporting  period. 

-  Upgrade  training  removal  data:  number 
of  persons  removed  temporarily,  number 
removed  permanently  and  percentage  of 
persons  removed  from  upgrade  training 

-  OJT  progress  evaluations  data:  number 
of  persons  due  for  supervisor  evaluation 
within  the  next  60  days,  number  of 
persons  due  for  commander  evaluation 
within  the  next  60  days  and  number  due 
final  evaluations  within  the  next  60  days 
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DATA  STORE:  Training  Report*  (continued) 


STRUCUTURE :  (continued) 

Data  ..Items  Descriptions 


FOR  WORKCENTER  TASK  COVERAGE  SUMMARY  REPORT: 


Report  Title 


Name  of  report. 


Report  Date  The  date  (DDMMMYYYY)  report  was  generated. 

Workcenter  Data  The  data  identifying  the  workcenter  for 

for  Report  which  the  report  is  generated.  Such  data 

will  include:  Component,  base,  unit, 
workcenter  and  AFS . 


Workcenter  Task 
Coverage  Data 
for  Report 


Data  which  are  aggregated  from  various  ATRs 
of  persons  assigned  to  the  workcenter  and 
from  OPTRs  to  which  the  persons  are  assigned. 
Report  data  has  been  calculated  to  generate 
the  report,  and  will  include: 

-  Number  of  persons  currently  assigned 
to  the  workcenter 


-  A  listing  of  the  tasks  contained  on  OPTRs 
which  apply  to  the  workcenter  being 
reported.  Data  for  each  task  listed 
includes: 


—  AFS 

—  Task  ID 

—  number  of  persons  who  are  required 
to  perform  the  task 

—  number  of  persons  who  are  qualified 
(certified)  on  the  task 

—  percent  of  persons  who  are  qualified 
(certified)  on  the  task 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

FOR  WORKCENTER  UPGRADE  TRAINING  ROSTER: 

This  roster  is  a  printout  of  the  Individual  Upgrade  Training 
Status  Report  for  each  person  assigned  to  the  respective 
workcenter.  Refer  to  data  items/descriptions  provided  for 
INDIVIDUAL  UPGRADE  STATUS  REPORT  included  within  this 
datastore. 


FOR  WORKCENTER  TRAINING  EVENT  STATUS  SUMMARY  REPORT: 


Report  Title 


Name  of  report. 


Report  Date 


The  date  (DDMMMYYYY)  report  was  generated. 


Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 


Workcenter  Data 
for  Report 


The  data  identifying  the  workcenter  for  which 
the  report  is  generated.  Such  data  will 
include:  Component,  base,  unit  and 
workcenter. 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

WORKCENTER  TRAINING  EVENT  STATUS  SUMMARY  REPORT  (continued) 

The  data  extracted  from  each  persons'  ATR 
for  persons  assigned  to  the  workcenter  being 
reported.  The  data  shall  be  aggregated 
and  calculated  to  provide  this  report,  and 
shall  include: 

-  Data  identifying  task  knowledge  training 
events  accomplished  during  the  reporting 
period  (number  scheduled,  number 
completed,  and  percent  of  events 
scheduled  which  were  completed) 

-  Data  identifying  task  knowledge 
evaluation  events  accomplished  during  the 
reporting  period  (number  scheduled, 
number  completed,  and  percent  of  events 
scheduled  which  were  completed) 

-  Data  identifying  task  performance 
training  events  accomplished  during  the 
reporting  period  (number  scheduled, 
number  completed,  and  percent  of  events 
scheduled  which  were  completed) 

-  Data  identifying  task  performance 
evaluation  events  accomplished  during  the 
reporting  period  (number  scheduled, 
number  completed,  and  percent  of  events 
scheduled  which  were  completed) 

-  Data  identifying  ancillary,  additional 
duty  and  contingency  task  training  events 
accomplished  during  the  reporting  period 
(number  scheduled,  number  completed  and 
percent  of  events  scheduled  which  were 
completed) 


Training  Event 
Status  Data  for 
Report 
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DATA  STORE:  Training  Report*  (continued) 


STRUCUTURE:  (continued) 

Data  Items  ggagEiB&igflg 

WORKCENTER  TRAINING  EVENT  STATUS  SUMMARY  REPORT  (continued) 

Data  identifying  the  Quality  Control  (QC) 
events  accomplished  during  the  reporting 
period  (number  scheduled,  number 
completed,  and  percent  of  events 
scheduled  which  were  completed) . 

Data  identifying  the  aggregation  of  all 
types  of  events  accomplished  during  the 
reporting  period  (total  events  scheduled, 
total  events  completed,  and  percent  of 
total  events  scheduled  which  were 
completed) . 


FOR  WORKCENTER  RECURRING  TRAINING  REQUIREMENTS  SUMMARY  REPORT: 
Report  Title  Name  of  report. 

Report  Date  The  date  (DDMMMYYYY)  report  was  generated. 


Workcenter  Data  The  data  identifying  the  workcenter  for  which 

for  Report  the  report  is  generated.  Such  data  will 

include:  Component,  base,  unit  and 
workcenter. 


! 
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DATA  STORE:  Training  Reports  (continued) 


4 


STRUCUTURE:  (continued) 


Data  items  Pegcriptisns 

WORKCENTER  RECURRING  TRAINING  REQUIREMENTS  SUMMARY  REPORT 


(con ' t) 


List  of  Training  Data  extracted  from  the  ITRs  of  individuals 
Requirements  for  assigned  to  duty  within  the  workcenter  for 
for  Report  which  the  report  applies.  For  each 

individual  requiring  task  or  other  training 
(ancillary,  additional  duty,  etc) ,  the 
following  data  is  listed: 

-  full  name 

-  SSAN 

-  AFS 

-  each  training  requirement  (identified  by 
Task  ID,  course  ID  or  course  name) 

-  the  current  status  (none,  in  progress, 
awaiting  refresher  training,  etc)  of  each 
requirement 


FOR  UNIT  POSITION  QUALIFICATION  STATUS  SUMMARY  REPORT: 

Same  data  items  as  for  the  Workcenter  Position  Qualfication 
Status  Summary  Report.  Each  applicable  workcenter 's  data  are 
further  aggregated  to  generate  the  report  for  a  given  unit. 


FOR  UNIT  UPGRADE  TRAINING  STATUS  SUMMARY  REPORT: 

Same  data  items  as  for  the  Workcenter  Upgrade  Training  Status 
Summary  Report.  Each  applicable  workcenter' s  data  are  further 
aggregated  to  generate  the  report  for  a  given  unit. 


FOR  UNIT  UPGRADE  TRAINING  ROSTER: 

Same  data  items  as  for  the  Workcenter  Upgrade  Training  Roster. 
Each  applicable  workcenter' s  data  are  further  aggregated  to 
generate  the  report  for  a  given  unit. 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

FOR  UNIT  EVALUATOR  PERFORMANCE  SUMMARY  REPORT: 

Report  Title  Name  of  report. 

Report  Date  The  date  (DDMMMYYYY)  report  was  generated. 

Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 

Unit  Data  The  data  identifying  the  unit  for  which 

for  Report  the  report  is  generated.  Such  data  will 

include:  Component,  base,  unit,  workcenter 
and  AFS . 

Unit  Evaluator  Data  which  are  extracted  from  each 
Performance  Data  individual's  ATR  within  the  unit,  and  which 
for  Report  have  been  aggregated  and  calculated  for  the 

purpose  of  this  report.  Such  data  will 
include : 

-  Number  of  persons  qualified  as  evaluators 
(ie.,  certified  on  one  or  more  tasks) 

-  Number  of  tasks  qualified  to  evaluate 

(ie.,  at  least  one  person  is  certified) 

-  Number  of  task  evaluations  during  the 
reporting  period  (cumulative  number  of 
performance  evaluations,  cumulative 
number  of  performance  evaluations  failed, 
and  percentage  of  performance  evaluations 
failed) 

-  List  of  tasks  without  qualified 
evaluators  within  unit.  For  each  task': 
the  AFS,  Task  ID  and  task  statement  shall 
be  identified  on  the  list. 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

FOR  UNIT  TRAINING  EVENT  STATUS  SUMMARY  REPORT: 

Same  data  items  as  for  the  Workcenter  Training  Event  Status  . 
Summary  Report.  Each  applicable  workcenter' s  data  are  further 
aggregrated  to  generate  the  report  for  a  given  unit. 


FOR  UNIT  RECURRING  TRAINING  REQUIREMENTS  SUMMARY  REPORT: 

Same  data  items  as  for  the  Workcenter  Recurring  Training 
Requirements  Summary  Report.  Each  applicable  workcenter' s  data 
are  further  aggregrated  to  generate  the  report  for  a  given  unit. 


FOR  BASE  POSITION  QUALIFICATION  STATUS  SUMMARY  REPORT: 

Same  data  items  as  for  the  Workcenter/Unit  Position  Qualficatiori 
Status  Summary  Report.  Each  applicable  unit's  data  are  further 
aggregated  to  generate  the  report  for  a  given  base. 


FOR  BASE  UPGRADE  TRAINING  STATUS  SUMMARY  REPORT : 

Same  data  items  as  for  the  Workcenter/Unit  Upgrade  Training 
Status  Summary  Report.  Each  applicable  unit's  data  are  further 
aggregated  to  generate  the  report  for  a  given  base. 


FOR  BASE  EVALUATOR  PERFORMANCE  SUMMARY  REPORT: 

Same  data  items  as  for  the  Unit  Evaluator  Performance  Summary 
Report.  Each  applicable  unit's  data  are  further  aggregated  to 
generate  the  report  for  a  given  base. 
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DATA  STORE:  Training  Reports  (continued) 


STRUCUTURE:  (continued) 

Data  Items  Descriptions 

FOR  BASE  UPGRADE  TRAINING  ROSTER: 

Same  data  items  as  for  the  Unit  Upgrade  Training  Roster.  Each 
applicable  unit's  data  are  further  aggregated  to  generate  the 
report  for  a  given  base. 


FOR  BASE  TRAINING  EVENT  STATUS  SUMMARY  REPORT: 

Same  data  items  as  for  the  Workcenter/Unit  Training  Event  Status 
Summary  Report.  Each  applicable  unit's  data  are  further 
aggregated  to  generate  the  report  for  a  given  base.- 


FOR  BASE  RECURRING  TRAINING  REQUIREMENTS  SUMMARY  REPORT: 

Same  data  items  as  for  the  Workcenter/Unit  Recurring  Training 
Requirements  Summary  Report.  Each  applicable  unit's  data  are 
further  aggregated  to  generate  the  report  for  a  given  base. 


FOR  AD  HOC  REPORT: 


Report  Date  The  date  (DDMMMYYYY)  report  was  generated. 

Reporting  Period  The  inclusive  dates  which  the  report  covers. 

This  reporting  period  will  be  the  first  day 
of  the  preceding  month  through  the  last  day 
of  the  preceding  month. 


Ad  Hoc  Report  The  data  extracted,  sorted,  calculated  and  4 

Data  produced  for  a  given  ad  hoc  report.  Each 

ad  hoc  report  serves  a  particular  purpose, 
and  identifies  or  summarizes  specific 
data,  as  defined  by  the  requestor. 
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